The Storage Architect » USP-V

Hitachi Virtual Storage Platform: Optimised Architecture

Chris M Evans — Tue, 19 Oct 2010 07:42:43 +0000

This is a series of posts that cover the features of Hitachi’s new enterprise storage platform, the VSP (Virtual Storage Platform), also sold by HP as the P9500 array. Previous posts:

Hitachi Virtual Storage Platform: Disk Drive Architecture

Hitachi have modified the VSP array to provide significant performance improvements over the previous USP and USP V models. These changes may not be immediately significant but are worthy of discussion, as with any new technology, the devil is in the detail. As a background to this post, I suggest you read my previous discussion on Monolithic v Modular architectures.

USP V Ports

Hitachi USP High Level Architecture

The USP V array design (reproduced here in schematic format) consists of a central switched architecture with both shared memory and cache. Processing takes place on FEDs (Front-End Directors) and BEDs (back-end directors) that take care of host and disk I/O respectively.

Front-end processors are shared between port pairs, so for example on a 16-port front-end card there are 8 processors. It is typical to see scenarios where either the port bandwidth or the port processing power is fully utilised. For example, with very small blocksize I/O, a FED processor can be max’ed out. This requires the storage administrator to be aware of host I/O profiles and distribute workload accordingly, or risk performance impact as ports are loaded up with hosts. This fixed design isn’t desirable (in any array) and in fact when external storage virtualisation is used can result in the over-purchase of storage ports purely to ensure sufficient capacity is available; bear in mind that port pairs (i.e. the processor) can only have one identity, either host port, external port, or source/target port for replication.

VSP Ports

The VSP changes the FED/BED architecture by sharing the processors for use across all physical ports. This is shown schematically in the following diagram (reproduced from Hitachi presentation – I will be working on a better representation). Port processors are now on the Virtual Storage Directors (VSDs).

By decoupling the physical port and processor, the VSP provides the ability to maximise both port and processor utilisation; it enables the driving of more work through the array. This is a key benefit when virtualisating external storage, as the static nature of the previous design has been overcome, so more storage can be externalised.

VSP Architecture

There is also an additional benefit to abstracting ports and processors; as firmware/code upgrades are performed on the VSP, there is no need to worry about path failover. Typically, code upgrades temporarily disrupt I/O to hosts. This isn’t usually a problem as production environments dual-path all connections, however if connectivity problems exist, then code uploads can cause host outages. This doesn’t occur with the VSP.

Futures

Although processor sharing is a simple change, it has wider implications; array performance is improved and made more efficient and this improves the ability to manage variable workloads. However, the change also provides a basis for future enhancements that could be even more compelling. Virtualising processor workload introduces the ability to:

Implement QOS (Quality of Service) on I/O requests. Although basic server prioritisation occurs today, full virtualisation enables real QOS to be implemented on I/O workload in a much more granular fashion.
Implement Multi-tenancy. The USP V already offered workload segmentation through Storage Partitions (SLPRs) and Cache Partitions (CLPRs). The VSP has the ability to create virtual partitions that are also prioritised in terms of workload. This meets requirements of organisations to offer secure multi-tenancy without having to dedicate physical hardware.

Hitachi have moved forward by producing a platform that is more scalable and potentially offers future enhancements for highly scalable environments. Although the VSP is a step up from USP, looks to me like only a single step on an evolving journey.

Choosing Between Monolithic and Modular Architectures – Part I

Chris M Evans — Tue, 24 Aug 2010 18:19:32 +0000

The recent proposed acquisition of 3Par by Dell and/or HP has made me think a little more about the direction the storage industry is taking in terms of their storage array design architecture. Since storage arrays became a category of devices in their own right, we’ve seen the growth of the monolithic, sometimes called Enterprise storage array. Hu Yoshida discusses the subject on one of his recent blog posts. Looking at the wide range of storage devices, I’ve categorised arrays into the following groups:

Monolithic – this architecture is characterised by Hitachi USP, HP XP & EMC DMX and consists of a shared memory architecture and multiple redundant components.
Multi-Node – these devices use loosely coupled storage “nodes” with a high-speed interconnect providing scalability by adding extra nodes to the storage “cluster”. Products in this category include EMC VMAX and 3Par InServ.
Closely Coupled Dual Controller – this is the typical “modular” storage architecture characterised by IBM DS8000, EMC CLARiiON, Hitachi AMS and HP EVA.
Loosely Coupled Dual Controller - this category describes technology that are capable of device failover but aren’t closely coupled to enable individual LUN failover as the Closely Coupled model permits. This category is characterised by arrays such as Netapp FAS filers and Compellent Storage Center.
Single Controller – this category covers devices that act as standalone products, including SOHO storage devices such as the Iomega IX4 & Data Robotics Drobo series.

The above list isn’t exhaustive and it’s my own personal categorisation. There are many more vendors of technology than I’ve listed here. In addition, none of these lists qualify as “Enterprise” in their own right. The use of this term is a hotly debated subject.

Monolithic Architectures

EMC DMX High Level Architecture

Monolithic arrays use a shared cache architecture to connect front-end storage ports to back-end disk. This is shown clearly in the architecture diagrams shown here, representing the internal connectivity of the EMC DMX and Hitachi USP storage arrays. Each of the memory units is connected to each of the front-end directors and the back-end disk directors. Hitachi divide their cache into two halves for Clusters 1 & 2 in the array; EMC have up to eight cache modules. This architecture has positive and negative benefits; firstly having director connections connecting to all cache modules ensures resources aren’t fragmented; unless cache becomes completely exhausted there’s always connectivity to another cache module to process a user request. It also doesn’t matter on which port that request comes in; the cache module can process any request from any port to any back-end disk. This connectivity is also beneficial in terms of failure. If a cache module fails, for example, only the cache on that module is lost; in a fully deployed architecture the total cache would drop (by 1/8th in EMC’s case), but front and back-end connectivity would remain the same. With this model it is possible pair up storage ports and have a single LUN presented from 1 or more ports with no performance impact; the path length between a storage port and disk adaptor will always be the same.

This any-to-any model also has disadvantages. The connectivity is complex and therefore becomes expensive and requires overhead to manage and control the interaction between the various components. In addition, there’s a limit to the practical scalability of this architecture. With eight FE, BE and cache modules, there are 128 connections in place; (8x8x2). Adding a single cache module requires an additional 16 connections; similarly, adding more front or back-end directors requires more connectivity. Also monolithic arrays are based on custom components and custom design, increasing the ongoing maintenance and development costs for the hardware.

One other point to remember; front and back-end directors have their own processors. It is possible for the traffic across the directors to be unbalanced and for some processors to be more heavily utilised than others. I’ve seen a number of configurations where USP V FED ports are running at 100% processor utilisation due to to small block sizes. This means manual load balancing is required both in initial host placement and subsequently as traffic load increases. This fact is worth bearing in mind as we move to more highly virtualised environments as it is likely host port utilisation will start low and rise over time as more virtual machines are created.

Hitachi USP High Level Architecture

Now that the DMX platform has been put out to pasture in place of VMAX, it appears Hitachi are the only vendor continuing down the monolithic route. Next time I’ll discuss Multi-Node arrays and why they may (or may not) be a replacement for today’s monolithic devices.

Hitachi Bloggers Day: Day 0

Chris M Evans — Mon, 14 Jun 2010 11:10:11 +0000

Here I am again on the start of another vendor blogging day. As the title of this post suggests, this will be a trip to see Hitachi, or HDS (Hitachi Data Systems) if you prefer. The Bloggers Day is taking place over two days and is located in San Jose, just south of San Francisco in California. I’ve previously posted a list of the attendees, both from the blogging community and the Hitachi itself.

The IT world has changed since I first encountered Hitachi 7700E, 9900 and the recent USP/USP V ranges of Enterprise storage arrays that typify Hitachi’s hardware portfolio. Enterprise and Modular storage now take equal billing and many of the features that were once Enterprise-only have migrated to the modular products, blurring the lines between the two platforms. In addition Hitachi have offerings for NAS and object store. They also sell servers (believe it or not).

Is this a scenario that has occurred because of customer demand? Is it more likely that reliability and the virtualisation of everything means that the original premise of the enterprise array is no longer valid? I believe that we are seeing a gradual move from the network-centric data centre, via the storage-centric data centre to what will become the hypervisor-centric data centre and eventually application-centric cloud. Storage devices are no longer the place where data functionality is focused and it will be less so as time goes on. The logical place for data mobility will be in the hypervisor (at least the hypervisor will be the controlling entity) and storage will become a feature as networks are today. If this is right, then the concept of and need to differentiate Enterprise and Modular arrays will cease to exist.

The “simplification” of storage (and I say it in quotes deliberately, as storage remains complex) means vendors are having to stretch out into other technology areas. EMC set the trend by demonstrating great foresight in buying VMware. Cisco & HP have joined the unified computing club. So now has Hitachi, selling their own servers and promising us their view of unified computing.

This shift away from storage is the interesting highlight of HDS Bloggers Day. What exactly is the Hitachi offering in Unified Computing? How have they integrated their server hardware and storage lines? Most important, how is this all being managed through a harmonised management suite of tools?

Here’s a final thought. Is the acronym Hitachi Data Systems still valid? Perhaps the rebranding to just Hitachi is all part of a big plan.

I’m looking forward to catching up with people today (well, much later today) and will post more later.

Enterprise Computing: VPLEX – Write Back or Write Through?

Chris M Evans — Tue, 18 May 2010 09:44:28 +0000

A little discussion on Twitter between myself, @BasRaayman, @UltraSub and @esignoretti got me thinking about the evolution of VPLEX and the whole caching thing. It’s something I mentioned on one of my previous VPLEX posts and it could have significant impact on designing and implementing VPLEX solutions. Here’s the conundrum.

First of all, be aware that VPLEX is a write-through device. That means it doesn’t keep I/O in cache and confirm write-completion to the host; it waits until the data is secured on disk before confirming the write success. One the one hand this is a positive thing; the VPLEX clusters contain no data in cache that could become stale or in the event of a failure, not written to disk. In VPLEX-Metro where synchronous cross-site writes are permitted, it also makes sense from a simplicity point of view; everything is on physical disk before the host I/O is confirmed as successful, so there’s no recovery issues to worry about.

However, write through in a heterogenous environment might not be so good. Performance is directly connected to the storage layer. Storage design has to continue as before and two layers of complexity now have to be considered in order to assure best performance. By using the VPLEX layer to achieve replication, we also have a situation where the performance is entirely dependent on the time it takes to write data to the underlying storage supporting the virtual VPLEX LUNs – in all locations. It would be possible to create a very bad configuration with awful performance. It also means that diagnosing performance problems has another layer of complexity to wade through.

The trouble is, the concept of VPLEX effectively leans itself towards the need to have write-through I/O, as VPLEX is enabling multiple I/Os to the same data in multiple locations. If I/O was cached, there would be a significant increase in data inconsistency, if one of the VPLEX devices in the complex was lost, for example.

Of course, other VPLEX-like technology uses write-through or write-back (Hitachi USP V for example). I believe SVC also caches. Are they in a better situation? In some respects they are because USP V and SVC offer additional features like thin provisioning and snapshots, all of which are implemented at the virtualisation layer above the underlying storage.

What is everyone else’s opinion? Will we see more or less complexity with VPLEX?

Enterprise Computing: Barclays Bank Services Down Due to Storage Array Problems

Chris M Evans — Wed, 17 Jun 2009 13:11:05 +0000

It’s been reported in a few places that yesterday Barclays (UK bank) suffered an issue with a “disc array” (presumably they mean disk array) that took out their ATM and online banking systems. See the comments here and here.

Allegedly, Barclays now use USP-V arrays as their back-end storage devices, so presumably HDS USP-Vs were involved in yesterday’s problems. Systems seemed to have been down for a number of hours before normal service was resumed.

The first thing to say is that “stuff” happens. Hardware fails – arrays fail and it’s the same for all vendors. No vendor can ever claim that their hardware doesn’t fail once in a while. We all know that RAID is not infallible; in fact, it isn’t even necessary to have a hardware failure to experience service outage as many problems are caused by human error.

What surprises me with this story is the time Barclays appeared to take to recover from the original incident. If a storage array is supporting a number of critical applications including online banking and ATMs, then surely a high degree of resilience has been built in that caters for more than just simple hardware failures? Surely the data and servers supporting ATMs and the web are replicated (in real time) with automated clustered failover or similar technology?

We shouldn’t be focusing here on the technology that failed. We should be focusing on the process, design and support of the environment that wasn’t able to manage the hardware failure and “re-route” around the problem.

One other thought. I wonder if this problem would have been avoided with a bit of Hitachi HAM?

Enterprise Computing: USP-V – So Long And Thanks For All The Fish

Chris M Evans — Wed, 27 May 2009 17:19:19 +0000

So HDS’s announcement has turned out to be a complete disappointment. What it’s not:

It’s not new hardware.
It’s not providing more physical capacity.
It’s not providing dynamic tiering.
It’s not providing enhanced replication technology.

What is on offer is the ability to cluster USPs – a feature called Hitachi High Availability Manager. By cluster, this means connect two USP arrays together and have them work in an active-active configuration, with data replicated in either direction. This new feature seems to be answering only one problem - how do I get off my USP?

Back in 2004 (I think it was), when I first sat with HDS and had a presentation on USP, the key question (especially with virtualisation) was how to cope with the fact that a single USP is the SPOF (Single Point of Failure). People in the room viewed the USP like a network switch - subject to failure, requiring upgrade and so on. HDS was at pains to say that clustering USPs simply wasn’t necessary as the hardware was fully resilient. In fact, HDS have been at pains since then to offer 100% availability of data with the USP. So why is clustering a USP so necessary? How can we have higher than 100% availability? Let’s not forget, the level of availability of any system is based on the availability of the least resilient component, so if we have a USP (with 100.00% availability) virtualising external storage (with 99.9% availability), the weak point is the external storage. Clustering USP’s doesn’t improve this and never will. All HDS have answered with this offering is the migration issue. This isn’t a new feature.

Here are some questions that arise from the presentation and which weren’t answered:

How far apart can my USP cluster arrays be?
What’s the impact on latency?
How is data integrity maintained?
Does clustering also support TrueCopy, ShadowImage and COW Snapshots?
Does clustering change my array World Wide Names, and if so, how?
Can cluster arrays be at different microcode levels?
Can clusters be TrueCopy secondary devices, if so what replication links are required?
Do need specific multipathing software?

So, you may be asking why the odd title of this post. Have a look at here. It’s what the dolphins said before they left earth. Time to say goodbye to the USP-V as a player in the Enterprise array space.

Enterprise Computing: The New USP – A Dreary Storage Cluster?

Chris M Evans — Thu, 21 May 2009 21:36:15 +0000

Well, I truly hope not. Let me just explain what I’m talking about and things may become a bit clearer.

HDS have started their viral marketing for an announcement being made on 27th May. Claus Mikkelsen’s latest blog entry asks us to guess what the announcement will be, based on an acronym of REGRADES OUR CLASSY TREAT.

So, I think the answer being searched for is STORAGE ARRAYS CLUSTERED or CLUSTERED STORAGE ARRAYS,, depending how you want to put it – however the title of this post is my favourite alternative…

So let’s try and guess what HDS are going to announce. Looking back to when the USP was originally released, it was (and still is) sold on the concept of virtualising external storage, turning the USP, NSC55, USP-V and USP-VM into a storage controller, presenting cheaper storage products if they resided in the USP. Whilst this was great (and HDS extolled the virtues of how UVM could fix all our migration problems) there was one flaw – once you’ve virtualised using the USP, how to you non-disruptively get the USP out? I heard talk about ways in which multiple USPs could be clustered together to overcome this, but never saw it in practice. So, with this new release, is HDS finally solving the problem?

There are only two enterprise/monolithic products worth discussing, Symmetrix/DMX and USP/XP. The USP is running behind the latest EMC release, the V-Max, on a number of fronts:

Scalability – the V-Max now offers up to 8 nodes and 2400 drives. USP still sits at 1152.
Tiering – DMX and V-Max offer larger SSD drives – 200 & 400GB.
Performance – V-Max will offer FAST for better dynamic data placement.
Replication – V-Max implements new replication devices for disk-less three site replication.

So I’m really hoping HDS will give EMC something to worry about including:

Better Scalability – let’s have more than 1152 drives, we’ve been needing more than this for a while. Let’s have flexible clusters to grow arrays as required.
Dynamic Data Placement - let’s have something better than Volume Migrator. Start thinking of data at the sub-LUN level.
Dynamic Array Replacement – make it easy to remove one array, migrate external storage to another without impact.

Even better, announce something I’ve not even thought of and surprise us all.

Anyone got a suggestion as to what it should be called? USP-VC, USP-C?

USP-V does SATA

Chris M Evans — Mon, 05 Nov 2007 16:00:00 +0000

So the rumours (here and here) are true. Here’s the announcement to prove it. HDS are going to support 750GB SATA II drives in the USP.

This is an interesting position HDS are taking as thin provisioning will be able to take use of the enhanced drive capacity making the USP-V even more efficient than the DMX-4 with SATA drives.

However I do wonder whether HDS have been pushed into supporting SATA on the USP-V. I was under the impression that thin provisioning on external drives (the standard HDS line – use External SATA storage rather than configuring it within the USP itself) wasn’t going to be available in the initial release. Perhaps HDS had to support SATA in order to get best usage out of the thin provisioning option and to answer customer complaints about using thin provisioning with expensive storage.

What I’d like to see next is how HDS will now position internal versus external storage. At what point do externally connected SATA drives become more cost effective than internal ones? This announcement seems to muddy the waters from that perspective.

I imagine we will get an announcement from Hu explaining how logical it is and how it is all part of the ongoing strategy….

SPC

Chris M Evans — Wed, 03 Oct 2007 05:24:00 +0000

According to Wikipedia, lightning can travel at a speed of 100,000 MPH, however I think storage vendors are even faster than lightning when it comes to highlighting or dissing the competition.

Mere microseconds after reading Claus Mikkelsen’s blog on the USP-V SPC figures, there are posts from BarryW and BarryB, doing the highlighting and dissing respectively (I almost wrote respectfully there; that would have been a funny typo). BarryB must have no real work to do other than to write his blog, looking at the size of the posts he does!

Anyway. I’m not going to comment on the results because the others have done that enough already and I don’t think the details are that relevant. I think what’s more relevant is the stance EMC are taking in not providing figures for customers on the performance of their equipment. I can’t decide whether its a case of arrogance and therefore a feeling they don’t need to provide details because as BarryB says, the customer will buy anyway, or is it because the DMX will not match up to the performance of its competitors. I think it is a mixture of both.

EMC aren’t an array vendor any more and haven’t been for a long time. OK, it is the product they’re most remembered for historically, but their reach is now so wide and deep I think Symmetrix isn’t the focus of a lot of their attentions. If it was, DMX4 would not just scale by the GB, it would have more connectivity, more cache and EMC would have been the *leader* in the implementation of technology like thin provisioning, not the also ran.

On reflection, I think EMC should provide SPC figures. If DMX is better than the others and is “Simply the Best” prove it; bragging starts to sound hollow after a while.

DMX-4 Green or not?

Chris M Evans — Fri, 20 Jul 2007 21:28:00 +0000

After the recent EMC announcements on DMX-4, I promised I would look at the question of whether the new DMX-4 is really as green as it claims to be. I did some research and the results are quite interesting.

Firstly we need to set the boundaries. One of the hardest part of comparing hardware from different manufacturers is that they are intrinsically different (if they were too similar, the lawyers would be involved) and that makes it difficult to come up with a fair comparison. So, I’ve divided the comparisons into controller and disk array cabinets. Even this is difficult. The DMX has a central controller cabinet which contains only batteries, power supplies, interface boards and so on. The USP however uses half of the central controller for disks. The DMX has 240 drives per cabinet, however the USP has 256 disks per cabinet. This all needs to be taken into consideration when performing calculations.

Second, I want to explain my sources. I’ve tried to avoid the marketing figures for two reasons; firstly they usually refer to a fully configured system and secondly they don’t provide enough detail in order to break down power usage by cabinet and by component. This level of detail is necessary for a more exact comparison. So, for the USP and USP-V, I’m using HDS’s own power calculation spreadsheet. This is quite detailed, and allows each component in a configuration to be specified in the power calculation. For EMC, I’m using the DMX-3 Physical Planning Guide. I can’t find a DMX-4 Planning Guide yet, however the figures on the EMC website for DMX-4 are almost identical to those for DMX-3 and it’s as close as I can get.

DMX-3/4

The DMX figures are quite simple; the controller cabinet (fully loaded) takes 6.4KVA and a disk cabinet 6.1KVA. A fully configured controller cabinet has 24 controller slots, up to 8 global memory directors and 16 back and front-end director (FED) cards. A typical configuration would have eight 8-port FED cards and 8 BED cards connecting to all 4 disk quadrants. EMC quote the disk cabinet figures based on 10K drives. Looking at Seagate’s website and standard 10K 300GB FC drives, each requires 18W of power in “normal” operation, so 240 drives requires 4.32KVA. The difference between this figure and the EMC value will cover when drives are being driven harder and the power supplies and other components which need powering within a disk cabinet. We can therefore work on an assumption of 25.4W per drive on average.

Now the figures for the controller cabinet are interesting. Remember EMC have no drives in the controller cabinet so all the power is for controllers, charging batteries and cache. So all that 6.4KVA is dedicated to keeping the box running.

USP

The HDS power calculator spreadsheet is quite detailed. It allows specific details of cache, BEDS, FEDs and a mix of 73/144/300GB array groups. A full USP1100 configuration has 1152 drives, 6 FEDs, 4 BEDs and 256GB of cache. This full configuration draws 38.93KVA (slightly more than the quoted figure on the HDS website. Dropping off 64 array groups (an array cabinet) reduces the power requirement to 31.50 KVA or 7.43KVA for the whole cabinet. This means the controller cabinet draws 9.21KVA and in fact the spreadsheet shows that a full configuration minus disks draws 5.4KVA. The controller cabinet has up to 128 drives in it, which should translate to about 3.7KVA; this is consistent with the 9.21KVA drawn by a full controller cabinet. The 7.43KVA in a cabinet translates to 29W per drive, making the HDS “per drive” cost more expensive.

This is a lot of data, probably not well presented but it shows a number of things;

There’s an inescapable power draw per drive which can’t be avoided; this equates to about 20W per drive.
The controller frame needs about 6KVA and this varies only slightly depending on the number of controllers and cache.
The HDS controller is slightly more efficient than the EMC.
The HDS disk array is slightly less efficient than the EMC.

Neither vendor can really claim their product to be “green”. EMC are playing the green card by using their higher density drives. There’s no doubting that this does compute to a better capacity to power ratio, however these green power savings come at a cost; SATA drives are not fibre channel and not designed for 24/7 workloads. Whilst these drives provide increased capacity, they don’t provide the same level of performance and DMX systems are priced at a premium so you want to get the best bang for your buck. However, if EMC were to price a SATA-based DMX competitively, then the model is compelling, but surely that would take business away from Clariion. What’s more likely to happen is customers choosing to put some SATA drives into an array and therefore see only modest incremental power savings.

So what’s the future? Well, 2.5″ drives currently offer up to 146GB capacity at 10K and only half the power demands, which also translates into cooling savings. Is anyone using these in building arrays? Hybrid drives with more cache should allow drives to be spun down periodically, also saving power. Either way, these sorts of features shoudn’t come at the cost of the levels of performance and availability we see today.

One final note of interest…HDS are quoting figures for the USP-V. These show a 10% saving over the standard USP, despite the performance improvements…