The Storage Architect » vSphere

Amazon Web Services – VM Import

Chris M Evans — Thu, 16 Dec 2010 11:18:38 +0000

Amazon have announced the availability of a feature that allows vSphere virtual machines (as VMDKs) to be imported into AWS. Today the feature is restricted to systems running Windows 2008 Server Sp2 but will be expanded in the future. Now this concept sounds great but there’s a major drawback here – upload time. A lot of my VMs are multi-gigabyte in size. Uploading them to AWS wouldn’t be a trivial task and of course during this time the VM isn’t available. So “click to the cloud” is getting close but still has issues.

So what would have to be done to allow a VM to be down for the minimum amount of time? I think it wouldn’t be that difficult. Firstly a lot of the data within a VM is static – installation libaries and other files don’t change from day to day. There’s usually a core set of files which are updated continuously. Snapshots provide the facility to manage the upload process as follows:

Take a snapshot of a VM. Upload the snapshot to AWS. While the upload is taking place, the VM will have updates written to a new file, effectively keeping track of changes since the last update. Once the upload is completed, the VM can be shut down and the changes uploaded, reducing the down time to the upload of only changed data. Alternatively another snapshot can be taken and this data uploaded, while new changes are recorded to a third file. Theoretically the process can be repeated indefinitely, each time around the loop the upload time will be shorter and less data will be out of sync. Two things will affect the ability to minimise VM downtime; upload bandwidth to the cloud and rate of change of data.

I’ve signed up to use the vSphere plugin which will enable uploading of the VM image. Today uploading is only available by the API. It will be interesting to see how downtime will be minimised with both options.

VAAI – Not Just for Virtualisation?

Chris M Evans — Mon, 11 Oct 2010 08:54:41 +0000

Last month we saw the announcement of vSphere 4.1 and the support of new storage features centred around VAAI, the vSphere API for Array Integration. This introduced a number of array offload features including Full Copy, Block Zeroing and Hardware Assisted Locking. Now this new functionality, aimed at improving scalability is intended to offload CPU and memory intensive operations to the storage array, the place where logically it is best executed.

The new functions are implemented through changes to the SCSI protocol, introducing new SCSI commands that execute the desired operations. For example Full Copy uses the SCSI EXTENDED COPY command; Block Zeroing uses WRITE SAME. This of course leads thoughts towards two ideas; the use of these commands on supported arrays for non-virtualisation purposes and whether these features will be emulated by the hypervisor for virtual LUNs.

Non-virtualisation Usage

The new SCSI commands are implemented through SPC-4 the latest specification of the SCSI protocol, still in draft. There’s no reason to assume that, if implemented by the storage vendor these commands couldn’t be used for other purposes. Here are some examples.

Secure Defragmentation – defragmenting a volume isn’t typically secure as the source block being re-organised isn’t zeroed out post-move. Secure defrag could ensure source block data is overwritten multiple times in a more secure fashion.
Virtual Defrag – defrag of thin volumes causes problems as the old block location for data being moved still looks like it is in use by the host. Virtual defrag could more easily mark an old block as free (for instance by writing binary zeros over it using WRITE SAME). This would enable defrag to work quickly but more efficiently.
Secure Delete – rather than rely on workarounds (for instance Windows sdelete), a secure delete function could ensure data is overwritten on the array, without involving host I/O.
Fast File Copy – This function is similar to what could be achieved with snapshots, but replicating at the file level and using the host to initiate the file copies.
LUN Ghost Copy – Replication of LUNs for host cloning in non-VM environments.
Offloaded/Fast Backup – Offloaded processing of backups and copying to the array to save host cycles. Note that extended copy enables replication to other SCSI devices that can include tape units.

Whilst all of these options sound good, there are also some possible security exposures. For instance, having looked at the detailed specification of EXTENDED COPY, I see no security controls to ensure I am permitted to copy the data I am requesting. For instance, I’m not required to copy my own LUN; I can copy any data to/from any LUN; that could enable me to copy data that I’m not permitted to, onto a LUN on which I have access; I could copy security details, and if I was really clever, I could overwrite a target LUN/block that contains a passwd file or other security information and so gain access to a machine/system. Hopefully this loophole has been investigated and I’ve not fully understood the protocol.

Summary

SPC-4 functionality provides both VM benefits (that have been coded for) and non-VM benefits (which may be something to use in the future). Security, as ever is a major issue and perhaps turning VAAI functions on without exploring this angle, could be an exposure not worth taking.

Netapp: The Inflexibility of Flexvols

Chris M Evans — Mon, 02 Aug 2010 07:34:52 +0000

I’ve just been reading up on Data ONTAP 8.0 as part of some ongoing work I’m doing. You may be aware that version 8.0 of the filer operating system brings two major features; 64-bit support and the (sort of) integration of the Spinnaker code to create the multi-node version of ONTAP (called cluster mode).

The move to 64-bit is an essential feature required to enable Netapp filers to scale. Currently both aggregates and Flexvols are limited to only 16TB. Plenty of people have complained this limit is far too low and it is one of the most restrictive scaling issues within ONTAP. By moving to 64-bit aggregates, scalability is increased dramatically (but perhaps not proportially as expected) to 100TB. Unfortunately things are not all rosy. For instance, although aggregates move up in size, Flexvols do not; they stay at a mere 16TB. There are a number of other things to consider that mean Flexvols are still not as flexible as they should be. For instance:

There is no option to move Flexvols between aggregates without taking an outage. Volumes can be copied (vol copy command) or cloned (vol clone command). As volumes get larger, taking extended outages to simply move data is unacceptable. Netapp needs to add the facility to transparently move the underlying location of a Flexvol without user impact.
Aggregate type (32-bit or 64-bit) is determined at creation time. This means a 32-bit aggregate, once created, cannot be expanded past 16TB. It also means that after a migration of an existing system to Data ONTAP 8.0, the aggregates can still not be expanded. If you’re hoping an upgrade will give you extra flexibility – it won’t. You will have to create new aggregates and migrate your data – which of course can’t be achieved without outage, as we’ve just discussed.
A system with 64-bit aggregates cannot be reverted to a version of Data ONTAP lower than 8.0. So, think long and hard after that upgrade about whether you need 64-bit aggregates. Once you create them, they are here to stay.
Aggregates can be increased in size by adding disk – but not reduced in size. So, there’s no flexibility in being able to temporarily increase an aggregate size, then reducing it when capacity requirements decrease (or when data has been moved elsewhere).

It’s disappointing that there are still significant restrictions in the use of so-called FlexVols with the new release of Data ONTAP 8.0 Fracturing the code into “7-mode” and “cluster-mode” doesn’t help – for instance the new vSphere VAAI extensions are not supported in Cluster Mode, so this isn’t a practical route to take in order to get the additional functionality. With the noise we hear from Netapp about virtualisation, the address space for the number and size of objects should be much larger than you wil never need. Unfortunately we are still pushing the logical limits which causes issues with utilisation and mobility, increasing operational cost.

On balance, I haven’t reviewed other NAS vendor’s products to do a full comparision here, however as Netapp are perceived to be the market leader, I’d expect more from them after nearly 20 years of development. There are other NAS products out there and I can’t help thinking that it’s only a matter of time before their market share starts to hit the Netapp bottom line.

Enterprise Computing: VMware Announce vSphere 4

Chris M Evans — Tue, 21 Apr 2009 21:12:16 +0000

In case you haven’t noticed, the next chapter in the story of the unstoppable juggernaut that is VMware is here. It’s called VMware vSphere 4 (dropping the Virtual Infrastructure moniker) but is still essentially the same as the previous VMware with incremental improvements. The full story of what private and public clouds will be seems a long way off, however in the short term there are improvements to storage which are worthy of discussion.

vStorage Thin Provisioning

Although it existed in VI3, Thin Provisioning is now fully integrated in vSphere 4. A VMDK can be allocated as thick or thin. Thin VMDKs are created initially with only 1MB of storage. As data is written to the VMDK, then storage blocks are created on physical storage. Thin VMDKs definitely save on storage; on my test VMware environment I use a template of each Windows platform. My Windows 2008 template is created with a default 40GB of storage; this actually uses around 5GB when cloned to a thin VMDK.

Here’s a (plagiarised) graphic which goes towards explaining how vStorage Thin Provisioning works. It also shows that vStorage TP can be used in conjunction with thin provisioning on the underlying storage array. Question: why do that? Well, firstly, the benefits of vStorage Thin Provisioning are obvious – don’t allocate storage you’re not using, especially when you’re creating tens if not hundreds of virtual machines. Thin provisioning on the storage array also makes sense. It allows you to create a large datastore and grow into it; having a large datastore in the first place reduces the need to move or re-allocate VMDKs once they are created (more on this later).

Improved iSCSI Initiator Efficiency

The ESX iSCSI initiator has been completely re-written. This is expected to result in higher throughput and lower CPU utilisation. This also seems like the first step (of likely many) towards enabling converged and unified storage connectivity for the hypervisor. Of course it’s also a implied admission that performance of iSCSI needed to be improved.

Dynamic Expansion of VMFS Volumes

A new feature called VMFS Volume Grow allows the expansion of a datastore on a VMFS LUN. If the underlying physical LUN is expanded then Volume Grow can be used to expand the datastore to make use of the new space. Enabling this dynamic expansion allows more flexibility in virtual storage design and these kinds of features are going to be essential in enabling always-on technology.

Enhanced Storage vMotion

Storage vMotion has been improved in a number of ways. Datastores can now be moved between LUNs from different protocols (Fibre Channel, iSCSI or NFS). Removal of this restriction gives more options for placing data on the right platform. It also negates the discussion about protocol; eventually the protocol will be irrelevant and so we see again a move towards commoditisation of the storage array. Storage vMotion performance has been improved and can manage thick to thin conversion at the same time as migrating.

Pluggable Storage Architecture

PSA enables vendors to provide their own multi-path drivers to supplement the native multi-path drive (NMP). Presumably for EMC this means a version of PowerPath for ESX. As we move forward with larger VMware deployments, then clever multipathing will be essential. If we are seeing VMware as the new mainframe, then multi-pathing will need to be the new EMIF (if you don’t know what that means, ask me and I might write a post on it). Future architecture will be storage, interconnect and processing power; storage will be plugged into the interconnect with many connections (USP and V-Max both now support up to 128 physical connections, 192 on a USP with restricted disk backend) and efficient multi-path algorithms will be needed to make most use of this.

Summary

So, there are things I’ve not mentioned (VMDirect path, paravirtualised SCSI) but this post is getting long now. vSphere is definitely addressing storage needs for the future. I can’t wait to get stuck in and see it in action.