The Storage Architect » usp

Hitachi Virtual Storage Platform: Optimised Architecture

Chris M Evans — Tue, 19 Oct 2010 07:42:43 +0000

This is a series of posts that cover the features of Hitachi’s new enterprise storage platform, the VSP (Virtual Storage Platform), also sold by HP as the P9500 array. Previous posts:

Hitachi Virtual Storage Platform: Disk Drive Architecture

Hitachi have modified the VSP array to provide significant performance improvements over the previous USP and USP V models. These changes may not be immediately significant but are worthy of discussion, as with any new technology, the devil is in the detail. As a background to this post, I suggest you read my previous discussion on Monolithic v Modular architectures.

USP V Ports

Hitachi USP High Level Architecture

The USP V array design (reproduced here in schematic format) consists of a central switched architecture with both shared memory and cache. Processing takes place on FEDs (Front-End Directors) and BEDs (back-end directors) that take care of host and disk I/O respectively.

Front-end processors are shared between port pairs, so for example on a 16-port front-end card there are 8 processors. It is typical to see scenarios where either the port bandwidth or the port processing power is fully utilised. For example, with very small blocksize I/O, a FED processor can be max’ed out. This requires the storage administrator to be aware of host I/O profiles and distribute workload accordingly, or risk performance impact as ports are loaded up with hosts. This fixed design isn’t desirable (in any array) and in fact when external storage virtualisation is used can result in the over-purchase of storage ports purely to ensure sufficient capacity is available; bear in mind that port pairs (i.e. the processor) can only have one identity, either host port, external port, or source/target port for replication.

VSP Ports

The VSP changes the FED/BED architecture by sharing the processors for use across all physical ports. This is shown schematically in the following diagram (reproduced from Hitachi presentation – I will be working on a better representation). Port processors are now on the Virtual Storage Directors (VSDs).

By decoupling the physical port and processor, the VSP provides the ability to maximise both port and processor utilisation; it enables the driving of more work through the array. This is a key benefit when virtualisating external storage, as the static nature of the previous design has been overcome, so more storage can be externalised.

VSP Architecture

There is also an additional benefit to abstracting ports and processors; as firmware/code upgrades are performed on the VSP, there is no need to worry about path failover. Typically, code upgrades temporarily disrupt I/O to hosts. This isn’t usually a problem as production environments dual-path all connections, however if connectivity problems exist, then code uploads can cause host outages. This doesn’t occur with the VSP.

Futures

Although processor sharing is a simple change, it has wider implications; array performance is improved and made more efficient and this improves the ability to manage variable workloads. However, the change also provides a basis for future enhancements that could be even more compelling. Virtualising processor workload introduces the ability to:

Implement QOS (Quality of Service) on I/O requests. Although basic server prioritisation occurs today, full virtualisation enables real QOS to be implemented on I/O workload in a much more granular fashion.
Implement Multi-tenancy. The USP V already offered workload segmentation through Storage Partitions (SLPRs) and Cache Partitions (CLPRs). The VSP has the ability to create virtual partitions that are also prioritised in terms of workload. This meets requirements of organisations to offer secure multi-tenancy without having to dedicate physical hardware.

Hitachi have moved forward by producing a platform that is more scalable and potentially offers future enhancements for highly scalable environments. Although the VSP is a step up from USP, looks to me like only a single step on an evolving journey.

Choosing Between Monolithic and Modular Architectures – Part I

Chris M Evans — Tue, 24 Aug 2010 18:19:32 +0000

The recent proposed acquisition of 3Par by Dell and/or HP has made me think a little more about the direction the storage industry is taking in terms of their storage array design architecture. Since storage arrays became a category of devices in their own right, we’ve seen the growth of the monolithic, sometimes called Enterprise storage array. Hu Yoshida discusses the subject on one of his recent blog posts. Looking at the wide range of storage devices, I’ve categorised arrays into the following groups:

Monolithic – this architecture is characterised by Hitachi USP, HP XP & EMC DMX and consists of a shared memory architecture and multiple redundant components.
Multi-Node – these devices use loosely coupled storage “nodes” with a high-speed interconnect providing scalability by adding extra nodes to the storage “cluster”. Products in this category include EMC VMAX and 3Par InServ.
Closely Coupled Dual Controller – this is the typical “modular” storage architecture characterised by IBM DS8000, EMC CLARiiON, Hitachi AMS and HP EVA.
Loosely Coupled Dual Controller - this category describes technology that are capable of device failover but aren’t closely coupled to enable individual LUN failover as the Closely Coupled model permits. This category is characterised by arrays such as Netapp FAS filers and Compellent Storage Center.
Single Controller – this category covers devices that act as standalone products, including SOHO storage devices such as the Iomega IX4 & Data Robotics Drobo series.

The above list isn’t exhaustive and it’s my own personal categorisation. There are many more vendors of technology than I’ve listed here. In addition, none of these lists qualify as “Enterprise” in their own right. The use of this term is a hotly debated subject.

Monolithic Architectures

EMC DMX High Level Architecture

Monolithic arrays use a shared cache architecture to connect front-end storage ports to back-end disk. This is shown clearly in the architecture diagrams shown here, representing the internal connectivity of the EMC DMX and Hitachi USP storage arrays. Each of the memory units is connected to each of the front-end directors and the back-end disk directors. Hitachi divide their cache into two halves for Clusters 1 & 2 in the array; EMC have up to eight cache modules. This architecture has positive and negative benefits; firstly having director connections connecting to all cache modules ensures resources aren’t fragmented; unless cache becomes completely exhausted there’s always connectivity to another cache module to process a user request. It also doesn’t matter on which port that request comes in; the cache module can process any request from any port to any back-end disk. This connectivity is also beneficial in terms of failure. If a cache module fails, for example, only the cache on that module is lost; in a fully deployed architecture the total cache would drop (by 1/8th in EMC’s case), but front and back-end connectivity would remain the same. With this model it is possible pair up storage ports and have a single LUN presented from 1 or more ports with no performance impact; the path length between a storage port and disk adaptor will always be the same.

This any-to-any model also has disadvantages. The connectivity is complex and therefore becomes expensive and requires overhead to manage and control the interaction between the various components. In addition, there’s a limit to the practical scalability of this architecture. With eight FE, BE and cache modules, there are 128 connections in place; (8x8x2). Adding a single cache module requires an additional 16 connections; similarly, adding more front or back-end directors requires more connectivity. Also monolithic arrays are based on custom components and custom design, increasing the ongoing maintenance and development costs for the hardware.

One other point to remember; front and back-end directors have their own processors. It is possible for the traffic across the directors to be unbalanced and for some processors to be more heavily utilised than others. I’ve seen a number of configurations where USP V FED ports are running at 100% processor utilisation due to to small block sizes. This means manual load balancing is required both in initial host placement and subsequently as traffic load increases. This fact is worth bearing in mind as we move to more highly virtualised environments as it is likely host port utilisation will start low and rise over time as more virtual machines are created.

Hitachi USP High Level Architecture

Now that the DMX platform has been put out to pasture in place of VMAX, it appears Hitachi are the only vendor continuing down the monolithic route. Next time I’ll discuss Multi-Node arrays and why they may (or may not) be a replacement for today’s monolithic devices.

Hitachi Bloggers Day: Day 0

Chris M Evans — Mon, 14 Jun 2010 11:10:11 +0000

Here I am again on the start of another vendor blogging day. As the title of this post suggests, this will be a trip to see Hitachi, or HDS (Hitachi Data Systems) if you prefer. The Bloggers Day is taking place over two days and is located in San Jose, just south of San Francisco in California. I’ve previously posted a list of the attendees, both from the blogging community and the Hitachi itself.

The IT world has changed since I first encountered Hitachi 7700E, 9900 and the recent USP/USP V ranges of Enterprise storage arrays that typify Hitachi’s hardware portfolio. Enterprise and Modular storage now take equal billing and many of the features that were once Enterprise-only have migrated to the modular products, blurring the lines between the two platforms. In addition Hitachi have offerings for NAS and object store. They also sell servers (believe it or not).

Is this a scenario that has occurred because of customer demand? Is it more likely that reliability and the virtualisation of everything means that the original premise of the enterprise array is no longer valid? I believe that we are seeing a gradual move from the network-centric data centre, via the storage-centric data centre to what will become the hypervisor-centric data centre and eventually application-centric cloud. Storage devices are no longer the place where data functionality is focused and it will be less so as time goes on. The logical place for data mobility will be in the hypervisor (at least the hypervisor will be the controlling entity) and storage will become a feature as networks are today. If this is right, then the concept of and need to differentiate Enterprise and Modular arrays will cease to exist.

The “simplification” of storage (and I say it in quotes deliberately, as storage remains complex) means vendors are having to stretch out into other technology areas. EMC set the trend by demonstrating great foresight in buying VMware. Cisco & HP have joined the unified computing club. So now has Hitachi, selling their own servers and promising us their view of unified computing.

This shift away from storage is the interesting highlight of HDS Bloggers Day. What exactly is the Hitachi offering in Unified Computing? How have they integrated their server hardware and storage lines? Most important, how is this all being managed through a harmonised management suite of tools?

Here’s a final thought. Is the acronym Hitachi Data Systems still valid? Perhaps the rebranding to just Hitachi is all part of a big plan.

I’m looking forward to catching up with people today (well, much later today) and will post more later.

Enterprise Computing: Why Thin Provisioning Is Not The Holy Grail for Utilisation

Chris M Evans — Thu, 04 Jun 2009 11:02:10 +0000

Thin Provisioning (Dynamic Provisioning, Virtual Provisioning, or whatever you prefer to call it) is being heavily touted as a method of reducing storage costs. Whilst at the outset it seems to provide some significant storage savings, it isn’t the answer for all our storage ills.

What is it?

Thin Provisioning (TP) is a way of reducing storage allocations by virtualising the storage LUN. Only the sectors of the LUN which have been written to are actually placed on physical disk. This has the benefit of reducing wastage, in instances where more storage is provisioned to a host than is actually needed. Look a the following figure. It shows five typical 10GB LUNs, allocated from an array. In a “normal” storage configuration, those LUNs would be allocated to a host and configured with a file system. Invariably, the file systems will never be run at 100% utilisation (just try it!) as this doesn’t work operationally and also because users typically order more storage than they actually require, for a many reasons. Typically, host volumes can be anywhere from 30-50% utilised and in an environment where the entire LUN is reserved out for the host, this results in a 50-70% wastage.

Now, contrast this to a Thin Provisioned model. Instead of dedicating the physical LUNs to a host, they now form a storage pool; only the data which has actually been written is stored onto disk. This has two benefits; either the storage pool can be allocated smaller than the theoretical capacity of the now virtual LUNs, or more LUNs can be created from the same size storage pool. Either way, the physical storage can be used much more efficiently and with much less waste.

There are some obvious negatives to the TP model. It is possible to over-provision LUNs and as data is written to them, exhaust the shared storage pool. This is Not A Good Thing and clearly requires additional management techniques to ensure this scenario doesn’t happen and sensible standards for layout and design to ensure a rogue host writing lots of data can’t impact other storage users.

The next problem with TP in this representation is the apparent concentration of risk and performance of many virtual LUNs to a smaller number of physical devices. In my example, the five LUNs have been stored on only three physical LUNs. This may represent a potential performance bottleneck and consequently vendors have catered for this in their implementations of TP. Rather than there being large chunks of storage provided from fixed volumes, TP is implemented using smaller blocks (or chunks) which are distributed across all disks in the pool. The third image visualises this method of allocation.

So each vendor’s implementation of TP uses a different block size. HDS use 42MB on the USP, EMC use 768KB on DMX, IBM allow a variable size from 32KB to 256KB on the SVC and 3Par use blocks of just 16KB. The reasons for this are many and varied and for legacy hardware are a reflection of the underlying hardware architecture.

Unfortunately, the file systems that are created on thin provisioned LUNs typically don’t have a matching block size structure. Windows NTFS for example, will use a maximum block size of only 4KB for large disks unless explicitly overriden by the user. The mismatch between the TP block size and the file system block size causes a major problem as data is created, amended and deleted over time on these systems. To understand why, we need to examine how file systems are created on disk.

The fourth graphic shows a snapshot from one of the logical drives in my desktop PC. This volume hasn’t been defragmented for nearly 6 months and consequently many of the files are fragmented and not stored on disk in contiguous blocks. Fragmentation is seen as a problem for physical disks as the head needs to move about frequently to retrieve fragmented files and that adds a delay to the read and write times to and from the device. In a SAN environment, fragmentation is less of an issue as the data is typically read and written through cache, negating most of the physical issues of moving disk heads. However fragmentation and thin provisioning don’t get along very well and here’s why.

The Problem of Fragmentation and TP

When files are first created on disk, they will occupy contiguous sections of space. If this data resides on TP LUNs, then a new block will be assigned to a virtual TP LUN as soon as a single filesystem block is created. For a Windows system using 4KB blocks on USP storage, this means 42MB each time. This isn’t a problem as the file continues to be expanded, however it is unlikely this file will end neatly on a 42MB boundary. As more files are created and deleted, each 42MB block will become partially populated with 4KB filesystem blocks, leaving “holes” in the filesystem which represent unused storage. Over time, a TP LUN will experience storage utilisation “creep” as new blocks are “touched” and therefore written onto physical disk. Even if data is deleted from an entire 42MB chunk, it won’t be released by the array as data is usually ”logically deleted” by the operating system. De-fragmenting a volume makes the utilisation creep issue worse; it writes to unused space in order to consolidate files. Once written, these new areas of physical disk space are never reclaimed.

So what’s the solution?

Fixing the TP Problem

Making TP useful requires a feature that is already available in the USP arrays as Zero Page Reclaim and 3Par arrays as Thin Built In. When an entire “empty” TP chunk is detected, it is automatically released by the system (in HDS’s case at the touch of a button). So, for example as fat LUNs are migrated to thin LUNs, unused space can be released.

This feature doesn’t help however with traditional file systems that don’t overwrite deleted data with binary zeros. I’d suggest two possibilities to cure this problem:

Secure Defrag. As defragmentation products re-allocate blocks, they should write binary zeros to the released space. Although this is time consuming, it would ensure deleted space could be reclaimed by the array.
Freespace Consolidation. File system free space is usually tracked by maintaining a chain of freespace blocks. Some defragmentation tools can consolidate this chain. It would be an easy fix to simply write binary zeros over each block as it is consolidated up.

One alternative solution from Symantec is to use their Volume Manager software, which is now “Thin Aware”. I’m slightly skeptical about this as a solution as it places requirements on the operating system to deploy software or patches just to make storage operate efficiently. It takes me back to Iceberg and IXFP….

Summary

So in summary, Thin Provisioning can be a Good Thing, however over time, it will lose its shine. We need fixes that allow deleted blocks of data to be consolidated and returned to the storage array for re-use. Then TP will deliver on what it promises.

Footnote

Incidentally, I’m surprised HDS haven’t made more noise about Zero Page Reclaim. It’s a TP feature that to my knowledge EMC haven’t got on DMX or V-Max.

Enterprise Computing: The New USP – A Dreary Storage Cluster?

Chris M Evans — Thu, 21 May 2009 21:36:15 +0000

Well, I truly hope not. Let me just explain what I’m talking about and things may become a bit clearer.

HDS have started their viral marketing for an announcement being made on 27th May. Claus Mikkelsen’s latest blog entry asks us to guess what the announcement will be, based on an acronym of REGRADES OUR CLASSY TREAT.

So, I think the answer being searched for is STORAGE ARRAYS CLUSTERED or CLUSTERED STORAGE ARRAYS,, depending how you want to put it – however the title of this post is my favourite alternative…

So let’s try and guess what HDS are going to announce. Looking back to when the USP was originally released, it was (and still is) sold on the concept of virtualising external storage, turning the USP, NSC55, USP-V and USP-VM into a storage controller, presenting cheaper storage products if they resided in the USP. Whilst this was great (and HDS extolled the virtues of how UVM could fix all our migration problems) there was one flaw – once you’ve virtualised using the USP, how to you non-disruptively get the USP out? I heard talk about ways in which multiple USPs could be clustered together to overcome this, but never saw it in practice. So, with this new release, is HDS finally solving the problem?

There are only two enterprise/monolithic products worth discussing, Symmetrix/DMX and USP/XP. The USP is running behind the latest EMC release, the V-Max, on a number of fronts:

Scalability – the V-Max now offers up to 8 nodes and 2400 drives. USP still sits at 1152.
Tiering – DMX and V-Max offer larger SSD drives – 200 & 400GB.
Performance – V-Max will offer FAST for better dynamic data placement.
Replication – V-Max implements new replication devices for disk-less three site replication.

So I’m really hoping HDS will give EMC something to worry about including:

Better Scalability – let’s have more than 1152 drives, we’ve been needing more than this for a while. Let’s have flexible clusters to grow arrays as required.
Dynamic Data Placement - let’s have something better than Volume Migrator. Start thinking of data at the sub-LUN level.
Dynamic Array Replacement – make it easy to remove one array, migrate external storage to another without impact.

Even better, announce something I’ve not even thought of and surprise us all.

Anyone got a suggestion as to what it should be called? USP-VC, USP-C?

Betamax

Chris M Evans — Thu, 04 Dec 2008 21:13:00 +0000

In case you don’t always go back and look at comments (and let’s face it, it’s not easy to track them), then have a look at Tony’s comment from my post yesterday. It’s good to see a bit of banter going on and that’s what HDS could do with.

HDS have hardly developed a good blogging prowess, it’s more a case of “oh well, better do that” than taking a lead in new media.

Look at EMC.

There’s geeky leaky Storagezilla with his uber-technical posts and sneaky advance notice of EMC technology.

Next The Storage Anarchist with his ascerbic character and product assassinations of the competition.

And who can forget Chuck, EMC’s futurologist with his head in the cloud.

There’s others of course, filling in the technical detail. Apologies if I haven’t mentioned you by name. EMC have certainly grabbed Web 2.0 and given it a good shake.

Sadly HDS don’t seem to have the same enthusiasm for marketing to their customers. Blog posts are few and far between from the small slew of bloggers they have to date. Content is shallow and that’s a big problem.

We *all* know USP is faster than DMX. Anyone who’s had the products in their lab know exactly what I’m talking about. Unfortunately unless HDS make a song and dance about it, they’re going to be the Betamax of the Enterprise storage world.

Tony, keep the posts coming! Give is some real substance to beat up the competition with!!

Replacing the Virtualisation Component

Chris M Evans — Mon, 13 Oct 2008 12:00:00 +0000

There’s no doubting that storage virtualisation will prove to be a key component of IT architecture in the future. The overall benefit is to abstract the physical storage layer from servers either in the fabric, or through the use of a dedicated appliance or even in the array itself.

Over time, storage resources can be upgraded and replaced, potentially without any impact to the host. In fact, products such as USP from HDS are sold on the virtues of their migration features.

However at some stage the virtualisation platform itself needs to be replaced. So how do we do that?

The essential concept of virtual storage is the presentation of a virtual world wide name (WWN). Each WWN then provides virtual LUNs to the host. The virtualisation appliance manages the redirection of I/O to the physical device, which also includes responding to SCSI LUN information queries (like the size of the LUN).

Ultimately, the host believes the virtual WWN is the physical device and any change to the underlying storage is achieved without affecting this configuration. If the virtualisation appliance must be replaced, then the virtual WWN could change and this means host changes, negating the benefit of deploying a virtual infrastructure.

As an example, HDS and HP allow USP/XP arrays to re-present externally connected storage as if it is part of the array itself. LUNs can be moved between either physical storage medium (internal or external) without impact to the host. However, the WWN used by the host to access the storage is a WWN directly associated with the USP/XP array (and in fact decoding the WWN shows it is based on the WWN serial number). If the USP is to be replaced, then some method of moving the data to another physical array is needed. At the same time, the host->WWN relationship has to change. This is not easy to achieve without (a) an outage (b) host reconfiguration (c) using the host as the data mover.

There isn’t an easy solution to the issue of replacing the virtualisation tool. Stealing an idea from networking, it could be possible to provide a DNS style reference for the WWN with a “name server” to look up the actual physical WWN from the “DNS WWN”. Unfortunately whilst this would be relatively easy to implement (a name server already exists in Fibre Channel) the major problem would be maintaining data integrity as a DNS WWN entry is changed and reads/writes start occurring from a new device. What we’d need is a universal synchronous replicator to ensure all I/Os written to an array are also written to any other planned target WWN, so as the WWN DNS entry is changed, it can’t become live until a guaranteed synchronous mirror exists. I can’t see many vendors agreeing to open up their replication technology to enable this; perhaps they could offer an API for “replication lite” which was used solely for migration purposes while the main replication product does the big replication features.

In the short term, we’re going to have to accept that replacing the virtualisation engine is going to have some impact and just learn to work around it.

CU Free

Chris M Evans — Wed, 31 Oct 2007 11:31:00 +0000

Does anyone use CU Free? Here’s the reason for my question.

I’ve just implemented a migration from a pair of 9980V and 9970Vs to a single USP in one site and a 9970V and 9980V remaining in the other site. All of the MCU->RCU relationships (4 of them) are being used between the USP and the 99XXV boxes.

If I implement another USP in the site with the 9900′s I want to replicate from the existing USP. Will CU Free let me exceed the MCU->RCU restriction or is it just a helpful way of saving me having to enter all the paths for all the CU relationships I want to use? i.e. is the restriction of 4 still there regardless?

Virtualisation Update

Chris M Evans — Fri, 07 Sep 2007 19:31:00 +0000

Thanks to everyone who commented on the previous post relating to using virtualisation for DR. I’m looking forward to Barry’s more contemporaneous explanation of the way SVC works.

I guess I should have said I understand Invista is stateless – but I didn’t – so thank’s to ‘zilla for pointing it out.

So here’s another issue. If SVC and USP cache the data (which I knew they did) then what happens if the virtualisation appliance fails? I’m not just thinking about a total failure but a partial failure or another issue which compromises the data in cache?

I was always worried that a problem with a USP virtualising solution was understanding what would happen if a failure occurred in the data path. Where is the data? What is the consistency position? A datacentre power down could be a good example. What is the data status as the equipment is powered back up?

Using Virtualisation for DR

Chris M Evans — Fri, 07 Sep 2007 05:47:00 +0000

It’s good to see virtualisation and the various products being discussed again at length. Here’s an idea I had some time ago for implementing remote replication by using virtualisation. I’d be interested to know whether it is possible (so far no-one from HDS can answer the question on whether USP/UVM can do this, but read on).

The virtualisation products make a virtue out of allowing heterogenous environments to be presented as a unified storage infrastructure. This even means carving LUNs/LDEVs presented from an array into consituent parts to make logical virtual volumes at the virtualisation level. Whilst this can be done, it isn’t a requirement and in fact HDS sell the USP virtualisation on the basis that you can virtualise an existing array through the USP without destroying the data, then use the USP to move the data to another physical LUN. Presumably the 1:1 mapping can be achieved on Invista and SVC (I see no reason why this wouldn’t be the case). Now, as the virtualisation layer simply acts as a host (in USP’s case a Windows one – not sure what the others emulate) then it is possible (but not usually desirable) to present storage which is being virtualised to both the virtual device and a local host, by using multiple paths from the external array.

If the virtualisation engine is placed in one site and the external storage in another, then the external storage could be configured to be accessed in the remote site by a DR server. See example 1.

Obviously this doesn’t gain much over a standard solution using TC/SRDF other than perhaps the ability to asynchronously write to the remote storage, making use of the cache in the virtualisation engine to provide good response times. So, the second picture shows using a USP as an example, a 3 datacentre configuration where there are two local USP’s providing replication between each other but the secondary LUNs in the “local DR site” are actually located on external storage in a remote datacentre. This configuration gives failover between the local site pair and also access to a third copy of data in a remote site (although technically, the third copy doesn’t actually exist).

Why do this? Well, if you have two closely sited locations with existing USPs where you want to retain synchronous replication and don’t want to pay for a 3rd copy of data then you get a poor man’s 3DC solution without paying for that third data copy.

Clearly there are some drawbacks; you are dependent on comms links to access the secondary copy of data and in a DR scenario performance may be poor. In addition, as the DR USP has to cache writes, it may not be able to destage them to the external storage in a timely fashion to prevent cache overrun due to the latency on writing to the remote external copy.

I think there’s one technical question which determines whether this solution is technically feasible and that is; how do virtualisation devices destage cached I/O to their external disks? There are two options I see; firstly they destage using an algorithm which minimises the amount of disk activity or they destage in order to ensure integrity of data on the external disk in case of a failure of the virtualisation hardware itself. I would hope the answer would be the latter rather than the former here, as if the virtualisation engine suffered some kind of hardware failure, I would want the data on disk to still have write order integrity. If this is the case then my designs presented here should mean that the remote copy of data would still be valid in case of loss of both local sites, albeit as an async copy slightly out of date.

Can IBM/HDS/EMC answer the question of integrity?