SCSI Reserve

First of all we need to take a step back and look at how LUN updates are serialised.  I/O integrity to a single LUN is achieved using the SCSI RESERVE command.  This allows an initiator (i.e. a host) to lock out a LUN from other updates.  This feature is especially necessary in clustered environments where a single LUN needs to be owned by a single cluster member during update.  Maintaining a reserve on a LUN prevents concurrent and conflicting updates from occurring (although strictly speaking updates can’t occur concurrently).  The “Reserve -> Update -> Release” process works well with environments where one or two hosts perform updates but in the virtual world where many hypervisors could be wanting to update large LUNs then the reserve process became a bottleneck.

SCSI reserve is essentially the same as the mainframe RESERVE macro, used to lock out a LUN to one specific address space and/or LPAR.  Unfortunately it’s rather a blunt tool in terms of managing data integrity and as volumes or LUNs increase in size, then the level of contention also increases, having a direct impact on performance and throughput.

The mainframe solution was to use Enqueues, essentially a “gentleman’s agreement” between the LPARs to share information on access at a more granular level.  Enqueues work on individual files and have a scope – they can be locked locally or globally.  This means a file can be reserved out within a single LPAR or across all LPARs sharing enqueue information.  LPARs implementing enqueues don’t then use RESERVE and so eliminate the performance overhead of constant volume reservation.

COMPARE and WRITE

VAAI within VMware uses the new COMPARE and WRITE command to perform a similar task to enqueues.  It enables data to be updated by one hypervisor by validating the contents are what was expected before the update.  The whole compare and write process has to happen as one operation (or atomically) to guarantee consistency.  The hypervisors confer with each other to maintain their “gentleman’s agreement” on who currently owns a virtual guest.

Virtual Machine Replication

Where does that leave us in terms of replicating virtual machines?  Well today replication is performed by the array, applying updates to the remote copy of a LUN in the order the updates were applied to the primary LUN.  What if an array manufacturer allowed both copies of a LUN to be read/write?  In this scenario hypervisors in two locations could update a single LUN over distance.  In order to maintain integrity, the COMPARE and WRITE command could be extended to include writing to the remote LUN as part of the single atomic operation.  The hypervisors then still retain their “gentleman’s agreement” on what’s being updated at any point in time.

Of course the above definition seems simple, but we have to consider latency and as distances increase, latency is likely to be the reason this solution breaks down.  Remote replication would need to be synchronous to guarantee integrity and so distance has a direct impact on performance.  Alternatively asynchronous replication could be used, with the understanding that data integrity isn’t completely guaranteed.

I’m looking forward to seeing how the vendors intend to tackle the LUN replication issue as it’s one we’ve all been waiting for, for a long time.

Firstly, let’s look at the VMware storage options.  You can choose to use block storage, in which case your LUN is formatted with the VMware proprietary VMFS file system.  Alternatively you can use NFS, in which case the NFS server manages the file system natively.  The VMFS option has a number of benefits; it allows VMware to tightly control the interaction of individual virtual machines and hypervisors, managing locking, thin provisioning and so on.  However there are also disadvantages.  Most storage arrays see LUNs as the smallest unit of presentable storage and perform operations (such as replication and cloning) at that level.  On non-virtualised environments multiple LUNs are usually presented to a host, so there’s no problem with those functions.  However with VMware the best practice is to use a smaller number of larger LUNs that encompass more than one virtual machine.  In effect this is a complete reverse of the physical host LUN model and can lead to issues when replicating or cloning on the array.

NFS provides a different approach.  NFS shares are presented to the hypervisor and virtual guests accessed simply as files.  NFS provides the locking mechanism and any additional functions, such as snapshots and replication.  As an extra bonus, the file system can still be accessed by traditional methods, making it possible to copy, edit and backup virtual machines away from the control of the hypervisor.  The trade-off between NFS and VMFS is seen as one of flexibility over performance but this is probably unfair and somewhat simplistic.

Where does VAAI come into this?  VAAI pushes some of the “heavy lifting” work required to clone virtual machines down to the array.  It also enables sub-LUN locking, improving performance.  In the discussion with Nigel he questioned why VMware were wanting to retain control via VMFS yet pushing functionality towards the array with VAAI.  I see this as a separation of control over execution.  VMware undoubtedly want to retain control of key storage functions within VMFS and the hypervisor.  If they do, they can charge for them.  It also makes sense for the hypervisor to be in control of tasks that directly affect the status of a running guest.  If VMware manages the replication process then data integrity can more easily be maintained.  However, copying data to and from the array through the hypervisor takes CPU cycles and network bandwidth.  It also doesn’t allow the array to execute in the most efficient manner.  By comparison, allowing the array to perform copy functions independently, data is not traversing the storage network, additional host cycles are not being used and the array can both schedule and execute the copy function much more efficiently as it is aware of the entire requirement, rather than seeing things in a sequential fashion.

Therefore we can see a dual benefit to VAAI; firstly the work is offloaded from the hypervisor, but second it is executed more efficiently.  In comparison to NFS, it seems that by implementing VAAI features, VMware are looking to retain even more control over the storage layer than before.

So what would have to be done to allow a VM to be down for the minimum amount of time?  I think it wouldn’t be that difficult.  Firstly a lot of the data within a VM is static – installation libaries and other files don’t change from day to day.  There’s usually a core set of files which are updated continuously.  Snapshots provide the facility to manage the upload process as follows:

Take a snapshot of a VM.  Upload the snapshot to AWS.  While the upload is taking place, the VM will have updates written to a new file, effectively keeping track of changes since the last update.  Once the upload is completed, the VM can be shut down and the changes uploaded, reducing the down time to the upload of only changed data.  Alternatively another snapshot can be taken and this data uploaded, while new changes are recorded to a third file.  Theoretically the process can be repeated indefinitely, each time around the loop the upload time will be shorter and less data will be out of sync.  Two things will affect the ability to minimise VM downtime; upload bandwidth to the cloud and rate of change of data.

I’ve signed up to use the vSphere plugin which will enable uploading of the VM image.  Today uploading is only available by the API.  It will be interesting to see how downtime will be minimised with both options.

• 56% of data on virtual systems is regularly backed up. This seems like a small number but perhaps the term “regularly” is one to consider here.  If environments are cloned and used for test, then perhaps there’s no need to back these environments up as they’re deleted and re-seeded as required.  It would be interesting to know how this figure breaks down by production and non-production environments.
• Only 20% of virtual environments protected by replication or failover technologies. This is a remarkable figure and implies a number of things; array-based replication is still hard to get right with virtualisation; users don’t consider virtual environments “production enough” to replicate them; but probably most important is that this implies there is still a lot of work to be done getting replication right.  Of course VMware with their VAAI initiative are looking to fix this problem and features such as svMotion help, but we know that there is a disconnect between LUN-based VMFS file stores and the granularity required to fail over individual virtual guests to remote locations.
• 60% of virtual environments are not covered in DR plans. The issue here could be similar to the first point above; most of these environments might not be production and so be re-seeded as required, however as virtualisation becomes the norm rather than the exception, DR will become an increasingly important consideration.
• 72% of organisations experience downtime from system upgrades & 70% experience downtime from power outages and failures. There’s always going to be a certain amount of “fat-finger” syndrome in system upgrades but I think these two statistics indicate that there’s a lack of failure planning going into designing infrastructures.  Yes, hardware and software will fail; it always does.  The skill is designing to this and building an infrastructure that meets requirements, including resiliency.

I’ve attached the study in it’s entirety to this post.  I’d be interested in anyone’s feedback on their experiences with the points raised; whether they feel they are valid or not.

Symantec 2010 Disaster Recovery Study

• VAAI is for SCSI. The current implementation supports SCSI, presumably with NFS support “any day now”.  The good thing about the way the VAAI APIs have been implemented is that they use a standard, well known protocol and the accompanying standards organisation in order to ensure correct adoption.  But what about for NFS?  Is there a standards body?  What API communication method will VMware use to talk to the storage device?  Who will support NFS out of the box?  I suspect the answer to the last question will be Netapp, Netapp and perhaps, Netapp.  I’d be worried that this means a proprietary interface will therefore be developed.
• Will VAAI work cross-platform? The Block Copy feature of VAAI is touted as a way to clone VM guests and that’s a useful feature.  But, that’s not a task you’re likely to perform many times during the day.  What’s more likely is that VAAI will enable improved storage vMotion performance by offloading the bulk data movement to the array.  All the examples I’ve seen talk about moving data in-array; that is between LUNs on the same storage device.  Big deal.  With a well laid out storage array, I shouldn’t need to do that.  I’m much more interested in moving data between arrays.  How will that work with VAAI?  SPC-4 of the SCSI standard enables EXTENDED COPY to copy between SCSI devices, but which vendors will bother to support it?  After all, who wants to let their customers more easily move off their storage?  Possibly more interesting is how they will support it.  EXTENDED COPY between different vendors effectively creates a cross-vendor replication offering that doesn’t just have to be used for VMware.  This opens up the possibility of using the same SCSI commands for doing multi-vendor data replication of traditional LUNs.
• Will VAAI work cross-protocol? Here’s an extension to the last train of thought; will VAAI work between NFS and VMFS disk formats?  At the moment I doubt it as block and file are like chalk and cheese.  There would need to be another layer of format conversion that understood both protocols to make this feature work.

I know VMware/EMC have openly discussed that the current VAAI implementation is only version 1 and we can expect new features coming up.  I’d like to know if they address any of the above….

The new functions are implemented through changes to the SCSI protocol, introducing new SCSI commands that execute the desired operations.  For example Full Copy uses the SCSI EXTENDED COPY command; Block Zeroing uses WRITE SAME.  This of course leads thoughts towards two ideas; the use of these commands on supported arrays for non-virtualisation purposes and whether these features will be emulated by the hypervisor for virtual LUNs.

Non-virtualisation Usage

The new SCSI commands are implemented through SPC-4 the latest specification of the SCSI protocol, still in draft.  There’s no reason to assume that, if implemented by the storage vendor these commands couldn’t be used for other purposes.  Here are some examples.

• Secure Defragmentation – defragmenting a volume isn’t typically secure as the source block being re-organised isn’t zeroed out post-move.  Secure defrag could ensure source block data is overwritten multiple times in a more secure fashion.
• Virtual Defrag – defrag of thin volumes causes problems as the old block location for data being moved still looks like it is in use by the host.  Virtual defrag could more easily mark an old block as free (for instance by writing binary zeros over it using WRITE SAME).  This would enable defrag to work quickly but more efficiently.
• Secure Delete – rather than rely on workarounds (for instance Windows sdelete), a secure delete function could ensure data is overwritten on the array, without involving host I/O.
• Fast File Copy – This function is similar to what could be achieved with snapshots, but replicating at the file level and using the host to initiate the file copies.
• LUN Ghost Copy – Replication of LUNs for host cloning in non-VM environments.
• Offloaded/Fast Backup – Offloaded processing of backups and copying to the array to save host cycles.  Note that extended copy enables replication to other SCSI devices that can include tape units.

Whilst all of these options sound good, there are also some possible security exposures.  For instance, having looked at the detailed specification of EXTENDED COPY, I see no security controls to ensure I am permitted to copy the data I am requesting.  For instance, I’m not required to copy my own LUN; I can copy any data to/from any LUN; that could enable me to copy data that I’m not permitted to, onto a LUN on which I have access; I could copy security details, and if I was really clever, I could overwrite a target LUN/block that contains a passwd file or other security information and so gain access to a machine/system.  Hopefully this loophole has been investigated and I’ve not fully understood the protocol.

Summary

SPC-4 functionality provides both VM benefits (that have been coded for) and non-VM benefits (which may be something to use in the future).  Security, as ever is a major issue and perhaps turning VAAI functions on without exploring this angle, could be an exposure not worth taking.

I asked Chad the question – what happens if customers have heterogeneous environments?  The answer from Chad was “Good luck to them” – a comment I take to mean – you’re on your own.  Now Chad did and will point to the fact that there are vendors out there who provide 3rd party solutions equivalent to UIM in environments that are not pure VCE (VMware, Cisco, EMC), however is that a fair direction for EMC to take?

Personally, I feel that EMC are effectively telling customers that they adopt the Vblock model (and UIM), or they will be left out in the cold.  It’s an example of further moves to lock end users into their proprietary stack of technology.  However although Chad intimates that alternatives to UIM exist, unfortunately they will never be as good as UIM for one simple reason:

VMware, Cisco and EMC own and understand the APIs used to drive their products.  They can (and I suspect probably do) retain certain documentation that enables them to have a competitive advantage over 3rd party vendors attempting to add equivalent functionality.  In addition, they can also make microcode and software changes to those APIs to provide any feature that might be needed to make UIM a better product than their competitors.  This ability always gives them the edge over any other vendor looking to offer similar functionality to UIM.

Of course, why shouldn’t EMC/VMware & Cisco do this?  After all, they have the right to do anything with their products that they choose and far be it from me to tell any vendor what their product & marketing strategies should be.  But it is worth looking back to the attempts by many companies to offer unified storage management tools; no hardware or software vendor has yet provided a single unified storage management tool that works across heterogeneous environments.  Customers still retain multiple products for storage provisioning, in many cases continuing to make use of CLIs rather than graphical interfaces.  So, the chances of there being a successful generic virtualisation orchestration product out there are slim to none.

So what is the solution?  For EMC, their answer would be to take Vblock and UIM.  But what happens if you take that route and build all your processes around UIM, then decide to change your storage vendor?  At that point you have nowhere to go other than to re-engineer your entire orchestration process, creating a very high cost and barrier to change.

Here’s one other point to ponder; in February, EMC sold off a number of the Ionix tools to VMware.  So if these products were intended to be part of vCenter, why is UIM coming from EMC?  There’s one easy answer to this; if VMware sold the product, their other partners (like Netapp) could push for integration of their products into UIM.  By selling UIM as an EMC tool, this support will never happen and so Netapp are forced to either write their own tools or partner with 3rd party vendors.  In this way the VCE coalition can continue to control their market.

So do you think vendor lock-in is a good thing?

Now that I’ve moved my main desktop to Mac, (new iMac 27″ with the i7 processor and 8GB of RAM) I decided to revisit desktop virtualisation to plug the gaps on the Windows tools that don’t have a direct Mac equivalent.  It was also a great opportunity to set up an environment to use Office 2010 more frequently.  I decided to move away from VMware and try out the free Virtual Box from Oracle (nee Sun).  So far I’ve been quite impressed.

Installation of the software was a breeze; I’ve included some screen shots of the installation process in the gallery at the end of this post.  I installed version 3.2.8 for OS X, which is a 74.5MB download.

Before diving into a “production” Windows 7 installation, I thought I would have a bit of fun seeing whether I could virtualise hypervisors.  First off was Hyper-V.  I had no problem with the installation but Windows wouldn’t let me add the Hyper-V feature after installation.  Next I tried ESXi, 4.0.0.  This time I was more successful, after setting on using RedHat Linux as the emulation type.  I also managed to get the installation to work with FreeBSD.  However despite installation working, I found it wasn’t possible to start a virtual machine; Virtual Box just hung and had to be force quitted.  I guess I shouldn’t have been too surprised by this behaviour, however I did think ESXi might work as older versions of VMware Workstation worked on processors without any virtualisation capabilities.

So back to my Windows 7 experiences.  I installed Windows 7 64-bit onto a new virtual machine, giving it 1GB of memory.  Once I had Office 2010 & Visio (2010) installed, I gave Visio a spin.  So far, I’ve found that Windows 7 in Virtual Box on Mac runs better than my previous native Windows 7 PC.  I suspect there are few reasons for this.

Firstly the i7 processor in the new Mac is way quicker than the previous machine spec (AMD Athlon 64 x2 Dual Core 4400+).  Second (and possibly more importantly) the graphics capability of the iMac are far superior and Virtual Box supports 3D and 2D Video acceleration.  3D support attempts to use the underlying 3D graphics hardware (if it exists).  I was fortunate that when I recently ordered my iMac the new models with enhanced graphics had just been released, so I have the ATI Radeon HD 5750 with 1GB of RAM.  This definitely improves the overall responsiveness of Windows 7 in this situation, even when not running full-screen, which most of the time it isn’t.

Overall I’ve found Virtual Box simpler and easier to use than VMware Fusion.  Whether it’s as fully function or not remains to be seen as I use it further.  However for free, it’s an excellent and worthy tool.

What is It?

What is Zimbra?  If you’re not familiar with the product, it’s effectively an email server with additional functionality, providing the sort of features you’d expect with Exchange or other similar messaging products.  Zimbra is delivered on Linux (and in the appliance specifically on Ubuntu), which may cause problems for certain customers familiar with the Microsoft route for managing email services.  The packing of any product as an appliance should theoretically take out some of the knowledge needed to deploy in a virtual environment, but can appliances really make application delivery that simple?

Installation

I filled in the online form and obtained a link to download the trial software, which can be delivered in OVF/OVA format or via a URL.  The download isn’t small at around 2GB and expands to around 18GB when installed.  I installed Zimbra in my vSphere 4.0 environment as you can see by following the screenshots in the gallery at the end of this post.  On initial power up the appliance asks the standard questions on licence agreements, data storage location and then performs a first time installation.  A configuration user password is then required and at this point the remainder of the configuration is continued via a web browser.  You can see from screenshot 6 that the browser URL is provided.  The installation defaults to a DHCP IP address, so this is the opportunity to both set a fixed IP and the host name.  After a couple of failed installation attempts I found it best to get the network settings and most importantly the host name set correctly here.  Host names need to be FQDN, otherwise the antispam and antivirus engines don’t work correctly.  Although I have a working knowledge of Linux and Ubuntu, I couldn’t manage to correct a wrongly coded host name after installation.  It was easier to perform a full appliance re-install than mess about correcting the problem.

Appliance?  Good or Bad?

There’s no question that an appliance is a simple way to perform an application installation.  Having everything pre-packaged and simply answering questions makes life much easier for that first install; I saw that with Zimbra as I re-installed a number of times.  However I see appliances as having only this one-off  benefit.  They are not an answer to not understanding the underlying operating system or application.  Ask yourself the following questions:

• How will I perform maintenance on the appliance?  You can guarantee that any appliance will immediately be out of date within weeks of creation.  Patches and updates will still need to be applied over time.
• How will I upgrade my appliance?  What happens if you want to move to a later version of the operating system or application?  Specific knowledge is still required.
• How will I backup my appliance/application?  Again, platform and application knowledge is required.

Of course one solution is to continually move to new, later versions of the same application appliance.  This requires knowledge of how the data is stored within the application and development of processes to migrate the data, which may be more complex than simply understanding and managing the underlying O/S.

Zimbra Evaluation

Once I’d sorted out the teething problems with host names, the Zimbra appliance was remarkably easy to deploy.  I now have it managing mail for my “thevirtualisationarchitect.com” domain.  The administration console is web-based and written Ajax and can be seen in screen shots 8 & 9.  Most features are pretty intuitive and there’s little need to revert to the CLI to perform administrative functions.  The user GUI is also web-based and can be seen in screen shot 10.  It is very similar to Outlook Web Access in look and feel.  As Zimbra supports IMAP, configuration to Entourage was a 30-second task, removing the need to use the web GUI at all.

I’ll continue to evaluate the software for the licence period after which I have a decision to make.  All of www.thevirtualisationarchitect.com sits in the cloud – the website itself is hosted on Amazon AWS, so it seems appropriate to keep web services virtual too.