I had a small discussion today with a few folks on Twitter that was a forerunner to a Wikibon “Peer Incite” discussion on whether RAID was still relevant or not (hence the title of this post).  It seems to me that perhaps the Wikibon discussion was a good way to [...] ]]> Portakabin Modular Construction

I had a small discussion today with a few folks on Twitter that was a forerunner to a Wikibon “Peer Incite” discussion on whether RAID was still relevant or not (hence the title of this post).  It seems to me that perhaps the Wikibon discussion was a good way to introduce a “RAID beating” technology to the market (and that’s a subject for another post), but regardless it may be worth reviewing where we are with RAID and it’s relevance in the future.

Background

IBM 3380 Model CJ2

The concept of RAID for disk devices was first discussed in a 1987 technical paper called “A Case for Redundant Arrays of Inexpensive Disks (RAID)” by David A Patterson, Garth Gibson and Randy H Katz from the University of California at Berkeley.  The premise of their paper was to compare the price and performance of SLEDs (Single Large Expensive Disks) versus cheaper alternatives in a redundant form and get improvements in performance, scalability and reliability for a lower cost.  The paper compares the price of inexpensive disks to the IBM 3380, one of the first disks I ever used.  These were monster devices the size of fridges, as you can see from this image I’ve borrowed from IBM’s archives (full link here).

I spend many hours recovering individual files from failed devices, having to recover the lost files from backups.  I’m not saying these devices were unreliable, but unfortunately individual file restores isn’t a scalable solution to managing continually increasing capacities.  To give you an idea of scale, I managed a mere 300GB of IBM 3380 & 3390 storage in one installation; what seems an incredibly small amount.

So, RAID promised to increase reliability significantly by providing a means to recover lost data.  RAID-1 is probably the simplest and easiest of the RAID levels to understand; write the data to two separate drives and if one fails, you can read/write from the other.  RAID-5 and RAID-6 offer distributed parity data recovery, with single and dual parity respectively.  This reduces the RAID overhead (RAID-1 was 100% more expensive than no RAID on the disk costs alone) while retaining the ability to recover.  However RAID is not infallible.  Double disk failures (where a second or third drive fails during recovery of the first failed drive) can occur, despite how small the chances seem.  This means RAID should be part of a portfolio of protection measures and not your only one.

Today we see RAID in even the smallest home storage devices, embedded as software options in operating systems and of course an mandatory component of enterprise storage arrays.  However, RAID suffering from potential scalability issues.

The Scalability Problem

Whilst RAID was good for the hard drives of 20 years ago, we now are starting to see issues with the RAID architecture in a number of areas.  These issues are being driven by the sheer increase in capacity we’ve seen for modern disk drives; an increase that hasn’t been matched by an equivalent improvement in performance and I/O throughput.  Here are some of the challenges:

• Capacity improvements have increased 100-fold between the IBM 3380 and the latest range of Savvio 15K drives (1.26GB to 146GB)
• Performance has only increased 50-fold in the same period
• Density (volume occupied by drives) has increased 2 million-fold in the same period
• Price has dropped to 1/27,000th of the cost

With storage arrays that can hold up to 2000 drives, we can see that I/O isn’t keeping up with capacity even for the fastest hard drives on the market today.  The problem is exacerbated when we look at high capacity SATA drives, currently pushing 3TB and set to go higher very quickly.  These drives have slower spin speeds and less throughput, meaning RAID rebuilds have to be counted in hours (and potentially days) rather than minutes.  This extended rebuild time has a number of implications:

• There is a greater risk of data loss, as rebuilds are taking substantially longer
• There is a greater performance impact as rebuilds impact host I/O capacity

There is one other problem that also can’t be ignored and that’s the chance of a non-recoverable bit read error – that is, the chance that your data cannot be re-read from disk.  This error is recoverable with RAID, but not if  RAID recovery is taking place, as this failed read may be essential to rebuild missing data.  As an example, the latest Seagate Constellation drives have a non-recoverable bit read error rate of 1 bit in 10E15 (in fact the failure is reading a whole sector).  A 1TB drive has 1E13 bits.  If we have to recover a disk by reading the whole RAID stripe (imagine it consists of 10 disks) then we have a 1 in 10 chance that we will be unable to recover some of that data.  This risk doubles to 1 in 5 with 2TB drives and so on.  As we reach 8-10TB drives, rebuilding the entire RAID group in this instance means we are almost always likely to fail to recover some data.  These values will change with different RAID group sizes and disk capacities but we’re pushing the technology close to the edge at this point.

The Solutions

We’re already seeing some workarounds to the RAID scalability problems.

• Block-based RAID – these architectures change the way in which RAID is implemented to a block rather than whole disk level.  So, if a drive has a partial failure, only the failed areas of the disk are rebuilt.  In addition, if the RAID group isn’t full, the system doesn’t spend time rebuilding white space.
• Failure Prediction – Storage arrays like Hitachi’s USP & VSP use SMART to predict when a drive looks likely to fail.  That drive is then “soft failed” and the data copied off it while it is still working.  This means data can simply be moved to another drive, rather than rebuilt from the other disks in the parity group.  This has a significant impact on improving recovery time and reducing the impact on performance, but can be more expensive in drive costs and maintenance.
• Data Distribution – The IBM XIV storage array distributes data across many drives.  This has the impact of dramatically improving the time taken to recover failed disks as all drives participate in recovery (and protection is only RAID-1).  Of course the tradeoffs are both the increase in cost and the still possible risk of a double disk failure, which will impact every single LUN on the system.

Whilst the above solutions are good, I believe we need to see more from the drive manufacturers themselves.  Ultimately the reason RAID has become a problem is due to reliability and RAID rebuild times.  We need a new approach to the way host I/O and rebuild I/O is prioritised and managed by a drive.  For instance, with the ability to put large amounts of flash into a drive, flash could be used as a repository for RAID reads.  As a drive executes normal read/write operations, it caches any data needed for requested RAID rebuilds into flash.  This is then made available to the RAID controller via a separate channel to perform the RAID rebuild.  Effectively data is rebuilt on a failed drive only as that data is read/written from the original drive; any unaccessed data is rebuilt as a low priority task.  If this approach was coupled with the ability to highlight a failing sector (and so recover that first) then reliability improves.  This idea is only one thought; I expect extending RAID’s useful life will follow a similar path to that of the increase in drive capacities; lots of incremental improvements that over time move things forward.

RAID isn’t dead, merely evolving to meet new challenges.  In 20 years’ time I suspect RAID will still exist but will be barely recognisable from the original Berkeley paper.

Portakabin Modular Construction

In the first post of this series, we discussed the basic hardware configuration.  This post will look at connectivity and RAID configurations supported by the NS4600.

A quick glance [...] ]]> This is a series of posts on the Promise SmartStor NS4600 home storage server.  Previous posts:

• Hardware Review: Promise SmartStor NS4600 – Part I

In the first post of this series, we discussed the basic hardware configuration.  This post will look at connectivity and RAID configurations supported by the NS4600.

A quick glance at the back of the unit provides a clue as to what connectivity exists.  See the first image in this post.  There is are 2x USB, 1x eSATA and 1x Ethernet ports available. The Gigabit Ethernet connection supports multiple host protocols, which we’ll discuss in more detail later.  The eSATA port provides connectivity to external devices for either backup of the NS4600 or backup of the external device to the NS4600.  The USB port also supports the same source and target backup functionality, meaning effectively all host connection protocols are IP-based.  The USB port also functions as a printer server for USB printers.

RAID Support

The NS4600 supports RAID levels 0, 1, 5 & 10, implemented by the onboard Promise PDC42819 SATA RAID Controller.  The required RAID level is specified at the time a volume is created; multiple logical volumes are supported on the NS4600 as long as there are sufficient disks available.  This means, for example, two RAID-1 volumes could be created or a RAID-5 volume with another RAID-0 volume could be established.  RAID settings are configured from the “RAID Management” option in the online GUI (PASM).  A number of example screenshots showing various RAID configurations are displayed in the gallery at the end of this post.

Volumes form the foundation of how data is presented from the NS4600.  They give the options to use RAID to manage the trade-off between capacity and performance.  For example, a single RAID-0 volume could be used for backups, while the main data is RAID-1 protected, with a final drive kept as spare.  Of course drives are hot-pluggable, so not all slots need to be initially populated.  This means drives can be added to a RAID group to increase capacity over time.  For instance two drives could be used to populate the NS4600 in the first instance.  This can then be expanded dynamically, changing the RAID level or adding additional capacity.

Although RAID is implemented in hardware, the options available are reasonably flexible in offering multiple dynamically expandable configurations.  However, I’d question whether traditional RAID implementations are the way forward in home storage devices.  Bear in mind that 2TB drives are becoming the norm and that means within 18 months to 2 years, 3/4TB and even 5TB drives will become commonplace.  As we move to much larger capacities, unrecoverable read errors become a real issue, so rather than recovering an entire disk, the ability to recover individual chunks of data is more preferable.  This is the methodology Data Robotics have implemented within their BeyondRAID technology.  Promise are playing the trade-off between rock-solid RAID-in-silicon versus RAID-in-software.  At the moment my money goes with software RAID and the enhanced flexibility it brings.

RAID Rebuild on NS4600

Now I like a bit of fun with RAID systems and one of my favourite tricks is to exchange RAID drives within an array.  So, on one of the NS4600′s, I powered it down, removed all the drives and powered it back up.  Fortunately, the device configuration isn’t stored on the removed disks and the NS4600 remained accessible, although protesting at the fact no drives were present.  After adding the drives back (in a random order) the NS4600 detected them and recovered the RAID sets and I was back in business.  I wouldn’t advise removing all the drives in normal practice, however if a chassis completely fails, then presumably the data can be recovered in another unit (although I haven’t tested this).

Volumes

NS4600 - Free Disks - Unconfigured Volumes

Volumes are the logical entity that are created when establishing the RAID configuration of the NS4600.  The screenshot below shows the NS4600 before any volumes have been created.  There are four volumes in the free pool.  The second screenshot shows two volumes that have been created from the four available drives in my test NS4600.  Volumes and the RAID set on which they are stored are a 1:1 relationship; a volume may not span a RAID set and a RAID set may not contain more than one volume.  This may seem a little restrictive, however as we’ll see later, file systems and iSCSI LUNs are

NS4600 Multiple Volumes

contained within a volume and therefore volumes should be thought of as providing specific RAID availability.

Volumes can be expanded; see the screenshot, which shows a volume in the process of being expanded.  This volume is being converted from RAID-1 to RAID-5.  RAID-0 volumes can be expanded to larger RAID-0 volumes or moved to RAID-1 or RAID-5 configurations.  Basically, volumes can be increased in size or moved to a higher level of RAID protection – but not down.

NS4600 Disk Formatting

The NS4600 has the concept of spares as opposed to free (unused) disks.  A hot spare can be used to dynamically rebuild a failed RAID group.  The screenshot shows a rebuild in place for a failed drive.  In this instance I’d left drive 4 as the hot spare and pulled drive 1 to simulate a failure.  The NS4600 automatically kicks off the rebuild and spare drive becomes part of the RAID group.  Now we see the second issue with the use of traditional RAID systems.  I simulated this RAID failure on RAID groups containing no data, yet the rebuild took hours due to the nature of the rebuild – a physical

NS4600 Raid Rebuild

drive recovery.  Contrast this with more progressive RAID systems where only the active data is copied, significantly reducing recovery times.  Of course the trade-off here is whether in a home system you would be rebuilding on a regular basis.  Chances are you wouldn’t but if a failure did occur, you would want it to occur as quickly as possible.  For the record, the total rebuild of a 2TB drive in a mirrored RAID-1 pair took 12 hours to complete with no other workload on the device.

In the next post I’ll look at the logical level of file systems and iSCSI LUNs.  Comments always welcome as usual.

Disclaimer: Promise have provided me with two NS4600 devices for this review.  These devices will be returned at the end of this period.  This is an independent review and has not been sponsored or paid for by Promise.

Promise Technology Inc have been manufacturing RAID controllers since 1988 and iSCSI storage systems since 2004.  In 2007, the company released the first of the SmartStor devices, the NS4300, a fully-functioned home NAS storage array.  That was followed [...] ]]> This is a series of posts on the Promise SmartStor NS4600 home storage server.

Background

Promise Technology Inc have been manufacturing RAID controllers since 1988 and iSCSI storage systems since 2004.  In 2007, the company released the first of the SmartStor devices, the NS4300, a fully-functioned home NAS storage array.  That was followed up in 2009 with the second generation NS4600.  I must admit I’m not at all familiar with their products, however Promise have provided me two NS4600 units for a short term evaluation.  The home NAS server market has become pretty competitive, with lots of features built into todays’ hardware.  What made the NS4600 interesting is the ability to backup between devices; something we will cover in these posts.  The ability to replicate to another unit will make it compelling for those advanced users who have large volumes of data to secure.

The NSx600 range are classed as high performance SOHO or home NAS devices and have the following specifications:

• Processor: Intel EP80579 600Mhz
• Memory: 256MB DDRII
• 1x GbE port
• 1x eSATA port
• 2x USB ports
• Four 3.5″ SATA drive slots
• Promise PDC42819 SATA RAID Controller

The device itself is pretty solid despite feeling quite light without the drives installed.  In terms of design, the unit looks quite attractive, with blue LEDs on the front, showing drive and network activity.  Drives are installed behind a single door in a horizontal fashion and are required to be mounted in small caddies.  These are screwed to the drive itself and aren’t large and provide the runners for correct insertion.  Drives are hot-pluggable while the unit is running.

At the rear of the unit, there are the network connections and power socket for the integrated power supply.  There’s also a power button for turning the unit on and off.  For cooling, there’s an integrated fan;

this kicks out a reasonable amount of heat when all four drive slots are occupied.  Although we’ll touch on software and management later, it’s worth mentioning here that the device (or enclosure as it is described) can be monitored through the built-in web server interface.  Screen shots are included at the end of this post and show the power, fan and temperature metrics being tracked.  I like this feature; it provides that extra level of information needed when doing problem determination.

Overall, the the NS4600 hardware is pretty cool.  In future posts, we’ll discuss software features and management.

Disclaimer: Promise have provided me with two NS4600 devices for this review.  These devices will be returned at the end of this period.  This is an independent review and has not been sponsored or paid for by Promise.

I’ve been messing about with some old hard drives this week and unusually for me, one is sounding decidedly sickly. I’ve never had a personal hard drive go on me (I guess I always upgrade/move on before it happens), but [...] ]]>

• Sabotage
• Server failure
• Catastrophic array failure
• Software bug
• Site failure
• User stupidity

If data is the lifeblood of your organisation then you *must* replicate it onto another online copy or at least onto a backup and have multiple copies in multiple locations.

If anyone out there is not sure they’re protecting their data properly – then give me a call!

EVA disks are placed in groups – usually recommended to be one single group unless there’s a compelling reason not to [...] ]]> Thanks to all those who posted in response to Understanding EVA earlier this week, especially Cleanur who added a lot of detail. Based on the additional knowledge, I’d summarise again:

• EVA disks are placed in groups – usually recommended to be one single group unless there’s a compelling reason not to (like different disk types e.g. FC/FATA).
• Disk groups are logically divided into Redundancy Storage Sets, which can be from 6-11 disks in size, depending on the number of disks in the group, but ideally 8 drives.
• Virtual LUNs are created across all disks in a group, however to minimise the risk of data loss from disk failure, equal slices of LUNs (called PSEGs) are created in each RSS with additional parity to recreate the data within the RSS if a disk failure occurs. PSEGs are 2MB in size.
• In the event of a drive failure, data is moved dynamically/automagically to spare space reserved on each remaining disk.

I’ve created a new diagram to show this relationship. The vRAID1 devices are pretty much as before, although now numbered as 1-1 & 1-2 to show the two mirrors of each PSEG. For vRAID5, there are 4 data and 1 parity PSEG, which initially hits RSS1, then RSS2 then back to RSS1 again. I haven’t shown it, but presumably the EVA does a calculation to ensure that the data resides evenly on each disk.

So here’s some maths on the numbers. There are many good links worth reading; try here and here. I’ve taken the simplest formula and churned the numbers on a 168-drive array with a realistic MTBF (mean time before failure) of 100,000 hours. Before people leap in and quote the manufacturers numbers that Seagate et al provide, which are higher figures, remember arrays will predictively fail a drive and in any case with temperature variation, heavy workload, manufacturing defects etc, the probability is lower than manufacturing figures (as Google have already pointed out).

I’ve also assumed a repair (i.e. replace) time of 8 hours, which seems reasonable for arrays unattended overnight. If disks are not grouped, then the MTTDL (mean time to data loss) is about 44553 hours, or just over five years. This is for a single array – imagine if you had 70-80 of them – the risk would be increased. Now, with the disks in groups of 8 (meaning that data will be written across only 8 disks at a time), the double disk failure becomes 1,062,925 hours or just over 121 years. This is without any parity.

Clearly grouping disks into RSSs does improve things and quite considerably so, even if no parity is implemented, so thumbs up to RSSs from a mathematical perspective. However if a double disk failure does occur then every LUN in the disk group is impacted as data is spread across the whole disk group. So it’s a case of very low probability, very high impact.

Mark & Marc commented on 3Par’s implementation being similar to EVA. I think XIV sounds similar too. I’ll do more investigation on this as I’d like to understand the implications of double disk failures on all array types.

It seems that I’ve returned to a flurry of acquisitions. Yesterday there was the heavily reported (on the blogosphere) purchase of XiV by IBM. Tony [...] ]]> The holidays are over and it’s back to work for me. In fact I returned yesterday; the break was good however it is also good to be back.

It seems that I’ve returned to a flurry of acquisitions. Yesterday there was the heavily reported (on the blogosphere) purchase of XiV by IBM. Tony Pearson gives a summary of the features on his post. One thing that interests me is the use of distributed writes across an entire array by creating 1MB blocks from (presumably) LUNs and filesystems. If a drive fails, then the data is still available on other disks in the system and spread across a great number rather than a single drive (RAID-1) or potentially a small number of drives (RAID5/6).

I’ve been trying to get my head around what this means. On the one hand it sounds like a real problem, as a double drive failure could impact a wide number of hosts; it all depends on how well the 1MB chunks are distributed. However maybe it isn’t that much of a problem as the issue only arises when both of the chunks that mirror a 1MB block both occur on failing drives. I would expect that as the number of physical drives increases then the impact of double failure reduces, as does the number of 1MB blocks affected. In addition, a drive may fail only in one area rather than on the whole device, so the affected blocks could be quite small; the remainder could be perfectly readable and be quickly moved. No doubt Moshe and the team have done the maths to know what the risk is and compared it to that of standard arrays and wouldn’t be selling the product if it was not inherently more safe.

The only other issue I can see is what market the product will slot into; Tony mentions that the product is not for structured data (although I guess it supports it) but was designed for unstructured data of large binary file types. So, why use RAID-1 compared to say a 14+2 RAID-6 configuration which would be much cheaper in terms of the disk cost? Presumably another selling point is performance, but I would expect the target data profile (medical, large binary objects) to be more sequential than random access and not be that impacted by using SATA.

I guess only time will tell. I look forward to seeing how things go.

The other purchase announced today was that of Onaro by Netapp. Onaro sell SANScreen, a tool to collect and analyse fibre channel SANs and to highlight configuration issues. Whilst I think it is a good product, I don’t see the fit with Netapp’s business in the NAS market (in fact I’m sure SANScreen doesn’t currently support NAS), so where’s the benefit here other than buying up a company which must be close to or is making money.

I wonder who will be bought tomorrow?

It’s interesting to see the failure rates quoted, anywhere from 1-3%, which on the face [...] ]]> Just read Robin Harris’ post at his new blog location; http://blogs.zdnet.com/storage/?p=116 and his comment on another blog discussing RAID. He quotes a VAR who has tracked disk failures and thinks RAID is an expensive luxury for desktops.

It’s interesting to see the failure rates quoted, anywhere from 1-3%, which on the face of it, seems low. However when its *your* disk that has failed and the data is irretrievable, there’s cold comfort to be had in failure rate statistics. I run RAID1 on my server; I have two 500GB SATA drives. Backing up that volume of data on a regular basis is a nightware without investing in a very expensive backup solution like LTO and it is a real disappointment to see tape hasn’t kept pace with disk in terms of the capacity/cost ratio.

So, I’m sticking with RAID. I augment it with disk-to-disk backups because, yes, you do have to cater for the d’oh factor of user errors or even dodgy software which corrupts files too, but RAID works for me and that’s all I need to worry about.