The Storage Architect » svc

Enterprise Computing: VPLEX – Write Back or Write Through?

Chris M Evans — Tue, 18 May 2010 09:44:28 +0000

A little discussion on Twitter between myself, @BasRaayman, @UltraSub and @esignoretti got me thinking about the evolution of VPLEX and the whole caching thing. It’s something I mentioned on one of my previous VPLEX posts and it could have significant impact on designing and implementing VPLEX solutions. Here’s the conundrum.

First of all, be aware that VPLEX is a write-through device. That means it doesn’t keep I/O in cache and confirm write-completion to the host; it waits until the data is secured on disk before confirming the write success. One the one hand this is a positive thing; the VPLEX clusters contain no data in cache that could become stale or in the event of a failure, not written to disk. In VPLEX-Metro where synchronous cross-site writes are permitted, it also makes sense from a simplicity point of view; everything is on physical disk before the host I/O is confirmed as successful, so there’s no recovery issues to worry about.

However, write through in a heterogenous environment might not be so good. Performance is directly connected to the storage layer. Storage design has to continue as before and two layers of complexity now have to be considered in order to assure best performance. By using the VPLEX layer to achieve replication, we also have a situation where the performance is entirely dependent on the time it takes to write data to the underlying storage supporting the virtual VPLEX LUNs – in all locations. It would be possible to create a very bad configuration with awful performance. It also means that diagnosing performance problems has another layer of complexity to wade through.

The trouble is, the concept of VPLEX effectively leans itself towards the need to have write-through I/O, as VPLEX is enabling multiple I/Os to the same data in multiple locations. If I/O was cached, there would be a significant increase in data inconsistency, if one of the VPLEX devices in the complex was lost, for example.

Of course, other VPLEX-like technology uses write-through or write-back (Hitachi USP V for example). I believe SVC also caches. Are they in a better situation? In some respects they are because USP V and SVC offer additional features like thin provisioning and snapshots, all of which are implemented at the virtualisation layer above the underlying storage.

What is everyone else’s opinion? Will we see more or less complexity with VPLEX?

Enterprise Computing: Storage Expo – It’s All Over

Chris M Evans — Tue, 20 Oct 2009 12:00:16 +0000

So Storage Expo is over for another year. As usual it was the mix of meeting old friends, finding out who had moved to work with who and working out how early to head out to the pub!

For me, there were a number of interesting take aways:

SVC 5.0 – lots of new stuff here that will form part of a new post at some stage. Probably the biggest change being the use of SSDs in the chassis of the SVC server itself.
Ocarina – being touted on the Bluearc stand. More investigation on the potential benefits required here.

Other than that the most obvious difference was the lack of EMC and Netapp on the stands. The whole show had the feeling of moving towards more midrange rather than enterprise customers. Next year there will be no separate Storage Expo but rather it will integrate into 360 IT and move to Earl’s Court from Olympia.

So a couple of funny (ish) stories. While wandering the floor on the Wednesday, I spotted Hu Yoshida sitting at the Hitachi stand. I introduced myself, but I don’t think he had any idea who I was. Perhaps it was the jet lag!

I managed hold out until 12:30pm before heading to the pub (Hand and Flower) for a beer. I was promptly accosted by HP, giving away free beer vouchers. Unfortunately, HP giving away free beer took me completely off guard and I forgot to use my free 3Par vouchers. Perhaps that was a good thing.

Have a look at the attached photos; the rack shows the Ocarina device. The other photo shows 3Par’s marketing banner in the pub. Perhaps not the best place to advertise a “keep thin” promotion!

Enterprise Computing: What Next For Virtualisation?

Chris M Evans — Tue, 15 Sep 2009 20:50:55 +0000

Earlier this month, Texas Memory Systems announced they had acquired the intellectual assets of Incipient, a company that produced SAN virtualisation hardware and software. With Incipient gone, EMC hardly bothering to mention Invista, what is the future of SAN LUN virtualisation?

I talked about Incipient last year, here and here when discussing the costs of performing migrations. As I said at the time, I couldn’t see how much of a saving deploying their iNSP would bring to the burdensome migration work we all have to manage on an ongoing basis. So there’s got to be a more compelling benefit out there for using virtualisation products. If there is, then what is it?

Excluding the defunct Invista, that leaves Hitachi with Universal Volume Manager (UVM) and IBM with SAN Volume Controller (SVC) still in the market place. From experience, I know UVM is a great product and surprise, I’ve commented on that recently too especially herewhere I reference the fact that Hitachi are offering UVM for free. Clearly, the drawback to UVM is that it is integrated into the array itself. When the NSC55 first came out, I heard rumours that it may be a diskless virtualisation “head” and although it can be deployed in that way, it isn’t sold as that. If Hitachi decided offer the USP VM or its successor as a diskless virtualisation controller, it would put them squarely in competition with SVC from IBM.

Earlier this year I was fortunate to have an invitation to meet Barry Whyte, “Master Inventor” and Performance Architect on the SVC product. You can find Barry’s blog hereif you’re already not subscribed to it. I highly recommend it especially for understanding the in’s and out’s of the SVC itself. During my trip I got to see some of the hardware used to do interoperability testing of SVC – with storage it virtualises as well as servers it connects to. It’s by no means a trivial task; there are 80 people in Hursley alone, working on development and testing of the product as well as a further 64 scattered around the globe. Obviously virtualising storage is a complex business and requires huge amounts of testing. I’d go as far as suggesting that the testing takes way more cycles than writing the code itself.

What’s all this got to do with the future of virtualisation? Well, I think it highlights what a complex process it is. Even though standards for interoperability exist, IBM (and presumably Hitachi, EMC and at one time Incipient) have to deal with complex interoperability issues and interleave that with additional features and functionality whilst guaranteeing data integrity. The slide taken from an SVC presentation deck gives you an idea of what’s involved. Thanks to Barry for permission to reproduce this.

Both Hitachi and IBM have been successful with a virtualisation product that doesn’t sit within the SAN fabric itself. This seems to me to be counter-intuitive as I’ve always thought the fabric was the right place for virtualisation. After all, every I/O leaving a host hits the fabric first and this naturally becomes the best place to route the I/O to its final destination, whether or not that is a “real” LUN or one created from a virtualisation product.

Perhaps SAN fabric virtualisation was simply too complex and costly to deploy. After all, recent history has told us that paying for a fabric-based virtualisation product is a non-starter otherwise we’d see more Invista and iNSP. Perhaps fabric-based virtualisation didn’t provide the feature set that mature IT organisations required from the technology. Either way, virtualisation in the fabric needs a rethink. Maybe FCoE provides/provided that opportunity?

Enterprise Computing: Data Migration Strategies – Part V

Chris M Evans — Wed, 22 Jul 2009 13:00:54 +0000

This is the final post in a series on Enterprise Data Migration Strategies. Previous posts:

Enterprise Computing: Data Migration Strategies – Part I

Enterprise Computing: Data Migration Strategies – Part II

Enterprise Computing: Data Migration Strategies Part III

Enterprise Computing: Data Migration Strategies – Part IV

Previously we’ve discussed how to plan, structure and organise migrations. In this post, I’ll touch on some of the tools which may be used to perform migration work.

One Size Does Not Fit All

First of all, its worth pointing out that no single solution fits all needs; migration methods are varied and the specific configuration in place demands the best solution at the time. Therefore it pays to have an arsenal of tools at your disposal and know how you’d use each one.

Array-Based Migration

Storage arrays already have many tools for performing migrations. These exist today for business purposes; remote replication to another array; local replication within an array using clones and snapshots. The benefit of using in-array technology is the migration work is taken away from the host and potentially can be executed within minimal customer interaction. On the negative side, most replication technologies which move data between arrays are product specific – i.e. SRDF on EMC DMX arrays isn’t compatible with HDS’ TrueCopy. This shouldn’t be a surprise because they are proprietary technologies which gain their advantage by being specifically coded and optimised to the storage platform itself. There are however tools like EMC’s Open Replicator which can move data between vendor/family technology. Open Replicator does have restrictions though – depending on the type/direction of replication, incremental copying isn’t available and requires a full copy sync to complete, potentially removing the benefit of using the tool altogether.

Virtualised Migration

Sitting slightly higher up the “storage stack”, it is possible to do migrations using a virtualisation technology sitting above (or integrated with) the storage array. For example, IBM’s SVC can be used to manage data migrations and sits above all storage arrays; HDS’ USP (equivalent to HP XP models) has a facility called Universal Volume Manager (UVM) which can perform the same work and is built into the array. Incipient have a solution called INSP (Incipient Network Storage Platform). If not already deployed, these tools will need an outage to be installed in the data path. Both impact the World Wide Name (WWN) the host sees, so host changes may also be necessary, depending on operating system. The benefit of these technologies, once installed, is that they allow data to be moved dynamically “under the covers” without involvement of the customer or work on the host server. As with all technologies, there are restrictions under certain circumstances and you should check with the product vendor for those. It may well be that you want to move the virtualisation tool at the end of the migration so another outage may also be required.

Fabric Migration

Moving slighly higher, we have the ability to perform data migrations in the storage fabric (SAN) itself. Example products include Brocade’s Data Migration Manager (DMM) and EMC’s InVista. Storage migration in-fabric requires the deployment of hardware in a SAN switch that intercepts I/O and redirects a second copy to another device. Potentially these devices can be installed in the data path without distruption but will require an outage to cut over to the new target volumes.

Host Migration

Finally at the top of the stack we have host-based migrations. Even at this level there are still a number of choices. If a Logical Volume Manager is installed (e.g. Veritas Foundation Suite/Volume Manager), then migrations can be performed using this software without host disruption. This is often a good choice of tool if the target devices are in a different array, if outages can’t be taken or if the LUNs are being re-organised or restructured. Unfortunately this also means having either host-access given to the storage teams (plus O/S knowledge to complete the work) or requiring the platform teams to perform the migration work. Both of these options may be a problem in certain organisational structures. One word of warning using LVMs – if LUNs are being replaced by using “evacuate” functionality (where a LUN at a time is swapped with another) then a potential data integrity problem exists, especially if the LUNs are also replicated. The risk occurs because data spans two arrays and if remotely replicated, then writes at the DR site might not be in integrity timestamp order. Failure in either array can result in an outage.

For mainframe customers, there’s the fantastic TDMF (also available in an Open Systems version)

If LVMs are not available, then good old-fashioned data copying is the order of the day. There are many tools to do this, too numerous to mention here, but be aware, that this method is likely to mean protracted downtime as storage shouldn’t be active and be accessed whilst it is being copied. It is also possible to migrate data within an application, again, there are too many options to mention here.

Hopefully this article provides a flavour of the migration tools out there. Please add comments or ping me if you’ve any specific tools you would like me to mention and I’ll add them on as a separate page.

LUN Stacker

Chris M Evans — Tue, 04 Nov 2008 13:00:00 +0000

A recent post from Martin “The Bod” Glassborow got me thinking about the whole process of LUN consolidation. I’ve done lots of migrations where people quake at the thought of changing the LUN size from one array to another. Now, I almost always want to change LUN sizes, as the vendor specific ones – 8.43GB/13.59GB etc are pretty painful and wasteful at the same time.

There’s another good reason to standardise on LUNs. If you’ve implemented a good dual-vendor strategy and sorted your firmware driver stack out, then you can position to take storage from any of your preferred vendors. There’s nothing better than having all of your vendors sweating on that next 500TB purchase when they know you take your storage from either or EMC/HDS/HP/IBM.

If LUNs and the I/O stack are all standardised, you can move data around too. The difficult part as alluded to in Martin’s post is achieving the restacking of data.

Here’s the problem; SAN storage is inherently block based and the underlying hardware has no idea of how you will lay out your data. Have a look at the following diagram. Each LUN from a SAN perspective is divided into blocks and each block has a logical block address. The array just services requests from the host for a block of data and reads/writes it on demand. It is the operating system which determines how the file system should be laid out on the underlying storage. Each volume will have a standard location (or standard method of calculating the location) for what was called the VTOC (Volume Table of Contents), also known as the FAT (File Allocation Table) in DOS and MFT (Master File Table) in NTFS. There are similar constructs for other O/S versions like Linux but I’m not 100% certain of the terminology so won’t risk the rath of getting it wrong.

The layout of data on a file system is not a trivial task. Apart from keeping track of files, there’s the requirement to keep track of free space and to be able to recreate the file index in the case of corruption, so some kind of journalling is likely to be implemented. There are also features such as compression, Single Instancing, Encryption, etc which all add to the mix of understanding exactly how file data is laid out on disk.

Now think of how multiple LUNs are currently connected together. This will be achieved with either a Volume Manager (like VxVM), supplied as a separate product, or a native LVM (logical volume manager). All of these tools will spread the “logical” volume across multiple LUNs and will format the LUN with information to enable the volume to be recreated if the LUNs are moved to another host. VxVM achieves this by having a private area on each LUN which contains metadata to rebuild the logical volume. Each LUN can be divided into sub-disks and then recombined into a logical volume, as shown in this diagram.

So a physical LUN from an array may contain a whole or partial segment of a host volume, including LVM metadata. Determining what part, whether all the parts are on this array (and where) is a tricky task – and we’re expecting that the transmission protocol (i.e. the fabric) can determine all of this information “on the fly” as it were.

My thought would be – why bother with a fabric-based consolidation tool? Products like VxVM provide a wide set of commands for volume migration, although not automated they certainly make the migration task more simple. I’ve seen some horrendous VxVM implementations, which would require some pretty impressive logic to be developed in order to understand how to deconstruct and reconstruct a volume. However life is not that simple, and host-based migrations aren’t always easy to execute on, so potentially a product would be commercially viable, even if the first implementation was an offline version which couldn’t cope with host I/O at the same time.

Funny, what’s required sounds a bit like a virtualisation product – perhaps the essence of this is already coded in SVC, UVM or Incipient?

Get the Balance Right

Chris M Evans — Fri, 31 Oct 2008 18:39:00 +0000

It’s not very often I side with one vendor or another however after BarryB’s recent post regarding “Benchmarketing” I feel obliged to comment. Have a read of Barry Whyte’s rebuttal too.

We see technology advancements because “concept” devices are used to drive innovation but don’t necessarily translate directly to end-user products. Look at the fashion industry – some of the most outrageous outfits are paraded down the catwalk but the same dress, coat, hat or whatever isn’t sold in the shops. Instead it influences the next fashion season.

Look at the motor industry – concept cars appear well before actual consumer products. We may laugh at some and marvel at others – take the Bugatti Veyron. It is well known that Volkswagen make a loss on each car produced, however what counters this is the publicity, the research, the kudos of being able to claim Veyron technology (disputably the fastest car in the world) is deployed in the standard VW range. Lexus is another good example of a brand created by Toyota to perform the same function. Much the same can be said for Formula 1.

Now, I’m not endorsing IBM per-se here, however I don’t see the harm with IBM marketing a “concept” piece of technology which could lead to innovation in the future. After all, IBM is well known for research of this kind; the disk drive and the tape drive spring to mind.

Even EMC’s own bloggers question whether EMC is known for innovation and other than Symmetrix, I can’t think of one thing I view as an EMC “idea”.

Anyway, ’nuff said. As previously offered – I would love to take the position of moderator in developing real world benchmarking – bring it on!!

Replacing the Virtualisation Component – II

Chris M Evans — Tue, 14 Oct 2008 12:19:00 +0000

Thanks to Chuck who pointed out to me SVC’s ability to move virtual WWNs between nodes during replacement. At some stage in the future I may get to play with SVC but I haven’t so far, so this feature eluded me. Question: is SVC the *only* block virtualisation appliance to offer this functionality and is it a seamless operation or does it require downtime?

How about InVista or Incipient (or any other vendor I may have missed off – we will assume USP doesn’t have the facility)? Answers on a ~~postcard~~ comment please.

IBM’s Storage Announcements

Chris M Evans — Mon, 08 Sep 2008 21:39:00 +0000

Today IBM made a slew of new product announcements. None of it was a surprise due to lots of pre-announcement leaks and the fact the IBM session in Montpelier was to a certain degree the formal announcement of products already mentioned in standard press releases.

I can only describe the new product releases as slow and steady with nothing radical or earth shattering. What it does do is consolidate IBM’s product range – slightly better tape libraries, slightly better tape drives; midrange SAN appliances, archiving appliances and a reconfirmation of the latest releases on SVC.

The release is here.

One point of note, IBM list their acquisitions in a PDF at the bottom of the release (get it directly here). IBM have acquired a lot of both hardware and software companies and this echoes somewhat of EMC a few years ago. What’s interesting is to see how IBM will integrate and harmonise these products, rather than sell them as a disparate set of technologies. Should be fun…

Incipient Revisited

Chris M Evans — Mon, 23 Jun 2008 07:48:00 +0000

You will remember that I recently posted a comment about migration costs, specifically with relation to Incipient. My view was (and still is) that the majority of migration costs come from preparatory and remedial work rather than execution of the migration. Well, Incipient asked for the right of reply and I had a call last week with Robert Infantino, their Marketing and Alliances Sr VP.

The $5000/TB figure they were quoting was an average they had seen in the industry for certain vendors’ professional services time to come in and perform the migration work on behalf of the customer. Incipient’s take was that they could provide their appliance/software expertise to provide the same service but at a significantly reduced cost (I won’t quote specific numbers here, but the number quoted was much lower than the equivalent cost from “a vendor”). So, I guess with clarification, it is more clear that Incipient were comparing the vendor costs versus their product costs and not including any internal customer costs (project management, preparation work etc) in the calculation. This seems a more appropriate comparison in my opinion.

Getting back to the vendor discussion, there’s a real issue here. If vendor X wants to sell you their latest technology, they need to accept and take the hit on helping with migration to their new array. This should be even more so where the vendor doesn’t change as this should be a “no brainer” and built into the technology.

In a world where hardware is becoming a commodity, one differentiator will be the vendor who can minimise the effort/cost and impact of migrating from one technology to another. Until then, products like SVC and those from Incipient will continue to have a market position – oh and humble consultants like yours truly!

SAN Virtual Appliances

Chris M Evans — Fri, 14 Sep 2007 05:45:00 +0000

LeftHand, FalconStor, Arkeia and Datacore all now offer VMware appliance versions of their products. I’m in the process of downloading them now and I’m hoping to install over the next few days and do some testing. I’ve previously mentioned some VM NAS products which I’ve installed but not reported back on. I’ll try to summarise all my findings together.

It seems that the market for virtual appliances (certainly in storage) is getting bigger. I think this is a good thing but I’m not sure that the virtualisation technology today provides capabilities to allow all vendors to virtualise their products. I suspect that the iSCSI brigade will get best benefit out of this wave of technology but fibre channel will not, as (from my experience) VM products don’t directly pass through fibre channel hardware to the VM guests (I’m aware of how RDM works in a VMware environment but I don’t think pass-through of target devices is sufficient).

Will IBM produce an SVC Virtual Appliance? I doubt it, but products such as Invista should be perfect candidates for virtualising as they don’t sit in the data path and the controller parts aren’t critical to performance. So EMC, show us your commitment to Invista and make 3.0 the virtual version!