The Storage Architect » dynamic provisioning

Enterprise Computing: Run My Storage At 60%? No Way!

Chris M Evans — Wed, 17 Mar 2010 19:52:53 +0000

Hu Yoshida has an interesting view on his recent post discussing storage utilisation rates. His concluding remark suggests running at a maximum of 60% utilisation – even with Dynamic Provisioning. Hu, you must be joking, right?

Point 1: I’ve paid for my 100% of storage and I’m going to use it. I don’t remember any vendor suggesting only paying 60% of their invoice and calling it quits. Granted, spending an inordinate amount of time to reach the 80%+ goal isn’t necessarily cost effective, however it can be achieved.

Point 2: It’s perfectly possible to run at 80% utilisation. It just takes some thought and planning. As Hu rightly points out, Dynamic Provisioning helps significantly towards this. The features of wide striping and thin provisioning mean storage can be more easily provisioned and requires less manual balancing.

Point 3: Achieving high utilisation isn’t all about technology. It’s also about process. That means Demand Planning, Capacity Planning, efficient processes for deployment of new hardware and for provisioning of customer requests. All this is possible without incurring additional expense.

So, don’t think 80%+ isn’t an achievable target. After all, it’s your money you’re wasting if you don’t try!

Enterprise Computing: Why Thin Provisioning Is Not The Holy Grail for Utilisation

Chris M Evans — Thu, 04 Jun 2009 11:02:10 +0000

Thin Provisioning (Dynamic Provisioning, Virtual Provisioning, or whatever you prefer to call it) is being heavily touted as a method of reducing storage costs. Whilst at the outset it seems to provide some significant storage savings, it isn’t the answer for all our storage ills.

What is it?

Thin Provisioning (TP) is a way of reducing storage allocations by virtualising the storage LUN. Only the sectors of the LUN which have been written to are actually placed on physical disk. This has the benefit of reducing wastage, in instances where more storage is provisioned to a host than is actually needed. Look a the following figure. It shows five typical 10GB LUNs, allocated from an array. In a “normal” storage configuration, those LUNs would be allocated to a host and configured with a file system. Invariably, the file systems will never be run at 100% utilisation (just try it!) as this doesn’t work operationally and also because users typically order more storage than they actually require, for a many reasons. Typically, host volumes can be anywhere from 30-50% utilised and in an environment where the entire LUN is reserved out for the host, this results in a 50-70% wastage.

Now, contrast this to a Thin Provisioned model. Instead of dedicating the physical LUNs to a host, they now form a storage pool; only the data which has actually been written is stored onto disk. This has two benefits; either the storage pool can be allocated smaller than the theoretical capacity of the now virtual LUNs, or more LUNs can be created from the same size storage pool. Either way, the physical storage can be used much more efficiently and with much less waste.

There are some obvious negatives to the TP model. It is possible to over-provision LUNs and as data is written to them, exhaust the shared storage pool. This is Not A Good Thing and clearly requires additional management techniques to ensure this scenario doesn’t happen and sensible standards for layout and design to ensure a rogue host writing lots of data can’t impact other storage users.

The next problem with TP in this representation is the apparent concentration of risk and performance of many virtual LUNs to a smaller number of physical devices. In my example, the five LUNs have been stored on only three physical LUNs. This may represent a potential performance bottleneck and consequently vendors have catered for this in their implementations of TP. Rather than there being large chunks of storage provided from fixed volumes, TP is implemented using smaller blocks (or chunks) which are distributed across all disks in the pool. The third image visualises this method of allocation.

So each vendor’s implementation of TP uses a different block size. HDS use 42MB on the USP, EMC use 768KB on DMX, IBM allow a variable size from 32KB to 256KB on the SVC and 3Par use blocks of just 16KB. The reasons for this are many and varied and for legacy hardware are a reflection of the underlying hardware architecture.

Unfortunately, the file systems that are created on thin provisioned LUNs typically don’t have a matching block size structure. Windows NTFS for example, will use a maximum block size of only 4KB for large disks unless explicitly overriden by the user. The mismatch between the TP block size and the file system block size causes a major problem as data is created, amended and deleted over time on these systems. To understand why, we need to examine how file systems are created on disk.

The fourth graphic shows a snapshot from one of the logical drives in my desktop PC. This volume hasn’t been defragmented for nearly 6 months and consequently many of the files are fragmented and not stored on disk in contiguous blocks. Fragmentation is seen as a problem for physical disks as the head needs to move about frequently to retrieve fragmented files and that adds a delay to the read and write times to and from the device. In a SAN environment, fragmentation is less of an issue as the data is typically read and written through cache, negating most of the physical issues of moving disk heads. However fragmentation and thin provisioning don’t get along very well and here’s why.

The Problem of Fragmentation and TP

When files are first created on disk, they will occupy contiguous sections of space. If this data resides on TP LUNs, then a new block will be assigned to a virtual TP LUN as soon as a single filesystem block is created. For a Windows system using 4KB blocks on USP storage, this means 42MB each time. This isn’t a problem as the file continues to be expanded, however it is unlikely this file will end neatly on a 42MB boundary. As more files are created and deleted, each 42MB block will become partially populated with 4KB filesystem blocks, leaving “holes” in the filesystem which represent unused storage. Over time, a TP LUN will experience storage utilisation “creep” as new blocks are “touched” and therefore written onto physical disk. Even if data is deleted from an entire 42MB chunk, it won’t be released by the array as data is usually ”logically deleted” by the operating system. De-fragmenting a volume makes the utilisation creep issue worse; it writes to unused space in order to consolidate files. Once written, these new areas of physical disk space are never reclaimed.

So what’s the solution?

Fixing the TP Problem

Making TP useful requires a feature that is already available in the USP arrays as Zero Page Reclaim and 3Par arrays as Thin Built In. When an entire “empty” TP chunk is detected, it is automatically released by the system (in HDS’s case at the touch of a button). So, for example as fat LUNs are migrated to thin LUNs, unused space can be released.

This feature doesn’t help however with traditional file systems that don’t overwrite deleted data with binary zeros. I’d suggest two possibilities to cure this problem:

Secure Defrag. As defragmentation products re-allocate blocks, they should write binary zeros to the released space. Although this is time consuming, it would ensure deleted space could be reclaimed by the array.
Freespace Consolidation. File system free space is usually tracked by maintaining a chain of freespace blocks. Some defragmentation tools can consolidate this chain. It would be an easy fix to simply write binary zeros over each block as it is consolidated up.

One alternative solution from Symantec is to use their Volume Manager software, which is now “Thin Aware”. I’m slightly skeptical about this as a solution as it places requirements on the operating system to deploy software or patches just to make storage operate efficiently. It takes me back to Iceberg and IXFP….

Summary

So in summary, Thin Provisioning can be a Good Thing, however over time, it will lose its shine. We need fixes that allow deleted blocks of data to be consolidated and returned to the storage array for re-use. Then TP will deliver on what it promises.

Footnote

Incidentally, I’m surprised HDS haven’t made more noise about Zero Page Reclaim. It’s a TP feature that to my knowledge EMC haven’t got on DMX or V-Max.

Enterprise Computing: HDS Switches On Virtualisation For Free

Chris M Evans — Wed, 22 Apr 2009 06:16:50 +0000

There’s no doubting HDS‘ Universal Volume Manager (UVM), aka external storage virtualisation is a cool product. I’ve used it many times – it does the job. However, the main drawback to using the product for me was always cost (I mentioned this only a few weeks ago on this post). Well, now that’s changed; until the end of this year, HDS are offering UVM for free. See the announcement here. HDS are calling it their “Switch It On” campaign.

OK, free isn’t quite free – there are a few caveats. Customers have to pay for maintenance, but other than that, there’s no charge for using UVM on current USP-V deployments. In addition, customers can “super-size” the offer and also get Hitachi Tiered Storage Manager (HTSM), Hitachi In-System Replication and limited access to Hitachi Dynamic Provisioning (HDP) all for free.

It is unlikely HDS are doing this out of some altruistic concern for their customers. Clearly they see benefits in driving more business by offering these licences at no cost. So how can customers benefit?

Traditional Approach

Traditionally, UVM is seen as a tool to manage the migration process or get data into a USP and there’s no question that UVM can be used for these purposes. Typically, here are some of the costs associated with migration:

Staff Resource Costs – migrations take time and effort to plan. The longer they take, the more cost is incurred with change control, planning, co-ordination with other teams and on actually performing the migration, which is typically performed out of hours.
Software Costs – migrations may require specific software products or tools (e.g UVM or TDMF).

Here are some of the constraints on migration work:

Limited downtime on servers.
Data to be migrated out of hours or within certain time frames.
No host tools (like volume managers) to perform migrations.
Data migrations are required to disparate locations.
Data integrity and synchronicity must be maintained (e.g. replicated data must be consistent if an outage occurred during the migration process.

The tradeoff with UVM has always been the cost of the UVM licence compared to the savings which could be made on the above issues by using the product. For example, UVM could be used to virtualise an entire array (or arrays) behind a USP-V in a single weekend. Consecutive weekends can then be used to migrate into the USP. This controlled approach (a) enables the storage managers to perform the migration, freeing up the host teams (b) allows the storage managers to load balance migrations into the USP-V (c) monitor performance as migrations occur (d) enable target servers to be immediately upgraded to new code/driver levels which may not have been supported previously on the older arrays.

Lateral Thinking

Getting UVM for free enables some more lateral thinking:

Example 1: Migrate out of the USP. There are lots of examples of moving data into a USP, but what about moving it out? If you’ve purchased a USP and it’s fully populated with disk and full, how do you save money? Well, one option is to get a UVM licence and migrate non-critical (and perhaps non-production) data to an externalised array at a lower cost. Previously this could only be commercially viable if the reduced cost of the external array covered the cost of the UVM licence. Now there’s no need to worry.
Example 2: Migrate in and De-Dupe. With UVM and HDP, external storage arrays can be migrated into the USP and placed on thin provisioned volumes. HDP Zero Page Reclaim removes the “empty” blocks of data during this process. For certain data profiles, this could provide signficant savings.
Example 3: A Free Datacentre Migration Tool. Imagine you’ve got to move data on arrays not currently replicated – but that data needs to reside in another location. If a pair of USPs are available, why not virtualise the source and target arrays and use the USP to move the data between the two sites? This could save purchasing replication licences for an array or resolve a problem where replication isn’t possible from certain hardware.
Example 4: Implement a 3DC Solution. How about using the USP-V as a three datacentre solution? Here’s how it works; two USP-Vs are used to replicate data synchronously between two local sites. The target devices are actually virtualised across a fabric to a third site using UVM. In this way, data can be moved to a third site without requiring a third copy of data.

With free software, the options are only limited by imagination. If HDS are prepared to offer software for free then there’s no excuse not to use it.

USP-V – bit of a let down?

Chris M Evans — Tue, 15 May 2007 21:04:00 +0000

It seems from the posts seen so far on the blogosphere that the USP release is causing a bit of a stir (10 points for stating the obvious I think). So, here’s my take on the announcements so far.

First of all, it’s called USP-V – presumably because of the “Massive 500-Percent Increase in Virtualized Storage Port Performance for External Storage”. I’m not sure what that means – possibly more on that later.

As previously pointed out, the USP-V doesn’t increase the number of disks it supports. It stays at 1152 and disappointingly the largest drive size is still 300GB and only 146GB for 15K drives. I assume HDS intends to suggest that customers should be using virtualisation to connect to lower cost, higher capacity storage. That’s a laudible suggestion and only works if the Universal Volume Manager licence is attractive to make it work. In my experience this is an expensive feature and unless you’re virtualising a shed-load of storage, then it probably isn’t cost effective.

There have been some capacity increases; the number of logical LUNs increases 4-fold. I think this has been needed for some time, especially if using virtualisation. 332TB with 16384 virtual LUNs meant an average of 20GB per LUN, obviously now it is only 4GB. Incidentally, the HDS website originally showed the wrong internal capacity here: http://www.hds.com/products/storage-systems/capacity.html, showing the USP-V figures the same as the base USP100. It’s now been corrected!

Front-end ports and back-end directors have been increased. For fibre-channel the increase is from 192 to 224 ports (presumably 12 to 14 boards) and back-end directors increase from a maximum of 4 to 8. I’m not sure why this is if the number of supportable drives hasn’t been increased (do HDS think 4 was insufficient or will be see a USP-V MKII with support for more drives?). Although these are theoretical maxima, the figures need to be taken with a pinch of salt. For example, the front-end ports are 16-port cards in the USP and there are 6 slots. This provides 96 ports, the next 96 are provided by stealing back-end directors (this is similar to DMX-3 – 64 ports maximum which can be increased to 80 by removing disk director cards). Surprisingly, throughput hasn’t been increased. Control bandwidth has, but not cache bandwidth. Does the control bandwidth increase provide the 500% increase in virtualisation throughput for external storage?

What about the “good stuff”? So, far, all I can see is Dynamic (thin) Provisioning and some enhancements to virtual partitions. The thin provisioning claims to create virtual LUNs and spread data across a wide number of array groups. I suspect this is simply an extension of the existing copy-on-write technology, which if it is, makes it hardly revolutionary.

I’d say the USP-V is an incremental change to the existing USP and not quite worthy of the fanfare and secrecy of the last 6 months. I’d like to see some more technical detail (HDS feel free to forward me whatever you like to contradict my opinion).

One other thought. I don’t think the DMX-3 was any less or more radical when it was released….