I’m in the process of migrating my existing physical infrastructure into a virtual environment. There’s nothing like “eating your own dog food” (there’s got to be a better expression than that) for testing out your beliefs on how technology should be implemented, so it’s only fitting I virtualise the IT infrastructure I rely on.
In summary, I run a “production” Windows AD domain, Exchange, IIS and file services. This is a typical scenario for many small businesses, with perhaps a few variations thrown in for good measure, such as SharePoint. I’m moving to a virtual environment that is built on ESXi v4.0, a custom design Intel-based server with 2x Quad Core Intel 5420 processors and 16GB of memory. Local hard drives include 2x SAS 7.2K 500GB drives and 2x 73GB SAS 15K drives.
Storage Choices
The first question I posed for my infrastructure is to decide on a storage platform. As this is a SOHO type deployment, I don’t have the luxury of a fibre channel SAN environment. I expect that many SMBs will feel the same and not want to commit to fibre channel deployments when the alternative options out there are low cost and probably more suitable. Therefore here are my choices:
Internal drives. I have over a terabyte of internal storage but unfortunately its not RAID protected. The disks are deployed on SAS connectors on the motherboard rather than through a dedicated RAID card. Although I could change the configuration and include a card, there are a number of issues; (a) increased cost (b) downtime to replace failed drives (c) physical intervention and downtime on the server to replace drives. My goal is to deploy this infrastructure to run 24×7, so having to take the system down to replace a drive is not an option. I also think that internal drives wouldn’t give me the scalability I might need.
iSCSI. In this environment, iSCSI is a great solution; there are lots of SMB iSCSI solutions on the marketplace today, offering low cost storage with simple operations. iSCSI is definitely a possibility.
NAS. Using NAS (more specifically NFS) for storage on VMware provides lots of flexibility. The management of the filesystem itself moves to the NAS hardware and so any VMware host files appear as standard files on the NAS device. When people see this, they usually get the “light bulb moment” as they realise the possibilities this kind of deployment offers. Whole virtual machines can be cloned, backed up or even simply copied as files around the infrastructure. NAS offers easy access to log and configuration files too.
So, for me the choice is NAS. I have iSCSI deployed, but for my production environments, NAS will be used to store my VMware hosts. This is for a variety of reasons, however there’s one additional benefit I haven’t discussed and that’s portability. Placing my vmdk’s on NAS means I can move them around at will. I can replicate them, I can back them up and I can move them easily to another NAS device without using ESXi as the data mover.
Stay tuned for more discussions as I evolve my setup.
My new DroboPro arrived this week and so far I’m less than impressed. As you know, I already have had a generation 1 Drobo for some time. It has been a great device, doing exactly what I wanted. After winning a second standard Drobo at Tech Field Day, I paid for the upgrade to the DroboPro in anticipation of a bigger and better machine. Here’s what I’ve managed to achieve and determine so far.
Hardware
The device itself is very different. First of all, the DroboPro accomodates up to 8 drives and so is orientated horizontally (otherwise it would fall over). At the back, the cabling supports Ethernet, USB and Firewire. Unfortunately the plugs are recessed and consequently removing the Ethernet cable is a pain as you can’t easily squeeze the retaining lug that releases the cable. The power supply is now integrated into the unit and there’s no separate PSU brick. This could present a problem if the power supply fails, as it isn’t obvious how (or even if) the integrated power supply could be replaced. I expect that power is now integrated to make the DroboPro more datacentre friendly. One last comment; the device is considerably more more noisy than its smaller brother, presumably because it has to cool twice as many hard drives.
Powering Up
Turning the device on is simple – there’s now an integrated on/off switch (hurrah). I decided to move the current drives in my Drobo into the ‘Pro and see how the upgrade process would work. This was when I hit the first non-intuitive issue. As the device is on its side, it isn’t easy to work out whether drives should be inserted with the top metallic cover pointing left or right. Turns out drives should be inserted with the top towards the status lights on the right. I had to look at the instructions for that crucial piece of information.
Once the disks were in, the DroboPro went through the standard boot process while I installed the Drobo Dashboard software from the installation CD. Here’s when I hit the next problem; I installed the software onto my MacBook as the DroboPro isn’t situated close to my desktop. Unfortunately the CD software crashed the Mac as soon as I started to configure the device.
Eventually, I managed to connect the DroboPro to a nearby Windows machine and deploy the latest version of Drobo Dashboard. The CD version also has a bug with Windows 2003 Server; I needed to download and install the latest 1.6.1 release.
Configuration
The original Drobo was pretty simple to configure; plug in the USB cable (or Firewire on Gen2) and away you go. DroboPro isn’t that simple, especially if you want to use iSCSI. Firstly, initial configuration needs to be made over USB. My device didn’t find my DHCP server, so I had to manually set the IP address once I was connected. Next I needed to create some volumes. This process although straightforward, seemed to completely ignore my volume names, leaving them all set as “Drobo”. I wasn’t able to subsequently rename the volumes once they had formatted.
Most irritating is the way the Drobo Dashboard interacts with the iSCSI initiator on Windows. Somehow, the software seems to dynamically add in the IP address of the DroboPro as a portal in the iSCSI definitions. If you remove it, the setting immediately gets put back in! I can’t find an easy way to force the Dashboard to stop binding the local machine to the iSCSI devices. This is an important issue because the VMware configuration guide requires the configuration of the DroboPro to be completed on a management PC separate to ESX and to detach or power off the PC before attempting to connect to ESX. I presume this happens because all the presented iSCSI LUNs are bound to a single iSCSI target and the first machine to log into that target locks it out for all other connections.
Summary
At this point, I have not yet connected my ESX server to the DroboPro (I ran out of time). However there are some fundamental issues which are either not possible or are not documented clearly enough. Here are my main issues:
USB and iSCSI Configuration – it isn’t clear whether the Drobo Dashboard can configure the DroboPro over either the USB connection or the iSCSI connection or whether USB connectivity needs to be retained for future configuration tasks. Be aware – unplugging the USB cable seems to cause the Drobo to immediately reboot.
Reduced Management – if the DroboPro is to be connected to an ESX server, then the documentation indicates the USB cable should be removed. What’s then not clear is how the ongoing status of the device can be managed. For example, at this point, where should the Drobo Dashboard be run from? The lack of a permanent management console means downtime to check the status of the device, to see the current occupancy levels and the effect of adding in additional capacity.
Lack of detailed Information – As yet, I haven’t managed to work out where or whether there are details provided on the utilisation of each configured LUN. The DroboPro allows for up to 16x 16TB LUNs, which is a huge volume of data (although ESX only allows iSCSI LUNs up to 2TB). I’d like to be able to work out which of those 16 volumes consumes the most physical compared to logical space, especially with ESX where the Drobo’s intelligent file system aware technology currently doesn’t apply.
Multi-Drobo Management. I have more than one Drobo device now (DroboPro, DroboShare & Drobo). The dashboard doesn’t appear to cater for multiple devices managed from the same machine.
The original Drobo was (and still is) a great device. I’d rather hoped that by adding the “Pro” moniker the DroboPro would be a step up in terms of features and management. As far as I can tell it isn’t. There are other devices on the market that provide better functionality with easier interfaces. To be fair, the ability to dynamically increase capacity over time is a compelling one, however I don’t think that feature alone justifies the choice of a DroboPro over other more feature rich storage devices already on the market.
I’d like to think I’ve visited a lot of customer sites over the years. Admittedly most of these are “enterprise” class with multi-terabyte if not petabyte quantities of storage. None of those customers have ever bothered deploying iSCSI as their storage protocol. Invariably block storage has been implemented using fibre channel and file using CFS or NFS. Somehow iSCSI just doesn’t seem to figure. I have a few thoughts on why…
Network versus Storage. There’s no doubt, Network and Storage teams get on about as well as cats and dogs. Although both support networking technologies, they are implemented fundamentally differently. In fact the only organisation I’ve seen that had the Network team managing fibre channel had implemented it like an IP network and it was a mess. As the iSCSI protocol means handing control of the physical transport layer to the network team, then the simpler option is to avoid using iSCSI in the first place.
It’s Too Easy. This may seem like a contradiction, but running a fibre channel network usually means managing a controlled environment. Nobody connects without permission, nobody gets access without being zoned in. Implementing iSCSI is simple and so inherently means less control.
There are no Standards. Fibre channel networks are great because you have to use expensive components and match everything against approved matrices or you don’t get support. iSCSI can be implemented using the cheapest NIC and virtual iSCSI targets. However, this ease of use also means there’s no vendor certification in the way there is with fibre channel. Who are you going to blame when things go wrong?
FCoE will Rule The World. Yes, Fibre Channel over Ethernet will be the One Storage Protocol to Rule Them All and replace fibre channel, iSCSI, AoE, NFS, CIFS, and any other protocol you care to name. OK, I’m being slightly sarcastic, but FCoE is set to harmonise the physical connect, leaving iSCSI redundant.
So what’s the future for iSCSI? We’ve seen the rise in popularity of home storage devices in recent years (think Iomega and Drobo). We’re now seeing these devices sporting Ethernet connectivity that supports iSCSI. With iSCSI Initiators (like the one in Windows) being totally ubiquitous, it’s a no-brainer to deploy iSCSI in home and small office environments. Maybe there is still a future for iSCSI after all in providing low-cost block storage for the consumer masses.
IanHF talked recently about LUN sizes and establishing standards across the enterprise. The choice of LUN size is a subject I’ve bored people with ad nauseum over the years and this is a good opportunity to go over it again.
Why Bother?
Ian’s post goes into more detail than the scope I intend to cover here. In fact, all I’m concerned with is LUN size. I don’t mind if it’s 1GB or 100Gb as a standard (well, I do, but that’s another post), all I care about is keeping a consistent LUN size across vendor technologies. Here’s why.
Host Based Migration
None standard LUN sizes mean host-based replication. It means using tools at the host level (like Veritas Storage Foundation) to restructure the data onto new LUNs. Worse, it could mean simply bulk copying of data at the host level, with outages to boot.
Good Advice
Here’s my advice. Pick a consistent LUN size. Choose one that works across all of your storage platforms at the block level, so when you want to move data by virtualisation or other vendor tools, you can achieve it without having to restructure the data.
I’m involved again with a client/customer I left about 3 years ago. At the time, I recommended taking the hit and migrating their disparate infrastructures to a consistent LUN size. They didn’t take my advice and they still have the same LUN size pain points they did 3 years ago. It means the migration process to move off old technology is more complex and expensive than it needs to be.
It’s amazing how such as simple change can be so impactful and how many organisations choose to ignore it.
Following on from my Personal Computing: Drobo Wierdness, I have received a replacement unit from Data Robotics. I thought for a change I’d produce a video and so here is my attempt to swap out my drives into the new unit. In the end I formatted the existing drives as four new NAS shares. So the unit is back and working. Enjoy.
The Gestalt IT Field Day was a great success in bringing together a mixture of delegates from varying discplines. Following the presentations from 3Par and Symantec, there was heated debate about the implementation of Thin Provisioning and the ability to reclaim released storage resources. This post covers the basic concepts of Thin Provisioning and more importantly how deleted resources can be recovered over time.
Thin Provisioning Primer
The underlying concept of thin provisioning is pretty simple; provide storage resources to those requesting it only as they need it.
Think of a standard ‘thick provisioned’ environment. As thick LUNs are created, the storage is assigned and mapped to that LUN to the full extent of the size requested. See, the first graphic, which shows a RAID group of four 5GB drives. I’ve assumed “RAID-0″ here for simplification, i.e. no RAID overhead. Each LUN (coloured separately) is made up from a 1GB slice of the available disks. Thick provisioning is great if the LUNs are all 100% allocated. In that instance, 100% of the available physical space is used up. However, it is never the case that 100% of a LUN is used and so wastage exists.
Look at the second graphic. This shows how thin provisioned LUNs work. As storage is requested by the LUN, the space is mapped to physical blocks of storage. In this example, none of the logical LUNs are fully utilised and so don’t consume their full theoretical capacity. This means that the pool of space can be over-subscribed and a sixth new LUN created. Obviously there’s no such thing as a free lunch or infinite storage resources and in this example if a further five blocks are requested then physical space would be exhausted. The next request for a new storage block would result in an error situation and this represents the main concern with over-subscribing thin provisioned volumes.
Now we get the concept of thin provisioning, there are a further two aspects to consider. Firstly, when we say a LUN isn’t 100% utilised, what to we mean? Second, how can deleted blocks be returned to our free physical pool?
As LUNs are presented to hosts, they are formatted with a file system, for example on Windows it’s NTFS; a VMware environment would use VMFS. The file system will have a standard layout which determines where the file index sits and the method in which files are allocated onto the disk. Have a look at the third graphic. This is a map of the C:\ drive for one of my servers. Each block represents approximately 22MB. You can clearly see the MFT (NTFS index) in the centre of the volume. A large percentage of the disk is unused. In a thin provisioned environment, storage would have been requested only for the blocks with valid data and in this way, a LUN can be less than 100% allocated.
OK, so what happens if I create some files then delete them on the file system? Most file systems remove a file by deleting the entry in the index rather than physically overwriting the file contents with binary zeros. This is quick and efficient (if not slightly unsafe security wise). The actual data isn’t overwritten and it is this ‘logical’ deletion that enables undelete utilities to work. The trouble is, most disk arrays are not file system aware and so can’t detect the logical deletion of a file. Those arrays that offer thin provisioning typically detect unwanted space by looking for blocks containing only ‘binary zeros’. This means simply deleting files will not release unused space back to the free block pool (except for one storage device I’ll discuss in more detail another time, that’s Drobo). Arrays which are capable of recovering unused space need to see data overwritten in order to recover it.
This (finally) brings us to the cookie analogy. Imagine cookies are my free pool blocks. There are a number of ways in which storage arrays operate in handling thin provisioning – different cookie monster personalities:
The Greedy Cookie Monster; grabs all the cookies he thinks he might eat, but never eats all of them and never returns any – this is the thick provisioning model.
The Selfish Cookie Monster; only grabs cookies as he gets hungry but if he doesn’t eat them immediately, doesn’t give them back – this is thin provisioning with no zero block reclaim. Eventually thin provisioning will become thick as all logical blocks in a LUN become mapped to physical storage.
The Nice Cookie Monster; takes the cookies as he gets hungry but only returns uneaten cookies if asked – this is thin provisioning with manual zero block reclaim. A manual process is required to zero out the unused space and to return it to the free pool.
The Saintly Cookie Monster; takes the cookes as he gest hungry and offers them back immediately he realises he can’t eat them – this is thin provisioning with automatic zero block or free space reclaim.
So, of the storage arrays out there offering thin provisioning, which fit the various Cookie Monster personality types? I’ll leave that for you to guess.
Day 2 of the Tech Field Day kicked off with a trip to Ocarina Networks. For those who don’t know (a) Ocarina offer a “data reduction” appliance (b) an ocarina is a small oval, china flute. I say data reduction as the Ocarina appliance uses a variety of methods for reducing data size, including compression and de-duplication. The main presenters for Ocarina were Carter George and the CTO Goutham (Gou) Rao. Rather than deliver death by PowerPoint, Gou chose to whiteboard his entire session, getting into some significant depth on the theory behind data reduction techniques. The whiteboard approach was very informative and popular with the delegates, generating lots of conversation.
Next up was Nirvanix. Nirvanix are a “cloud storage” company, operating in the same space as Amazon S3. After providing an overview of the company, Peter Pistek provided details of CloudNAS 2.0 a software offering which enables a Linux or Windows server to act as a NAS device, storing the data directly in Nirvanix’s cloud. This is a product I’ve reviewed in the past and previously I’ve not been favourable towards it. Expect to see my review of the 2.0 version in the near future.
Day 2 lunch was provided by W Curtis Preston and was followed by “Mr Backup” taking us through his new venture – Truth In IT, a paid-for community portal to assist clients in evaluating storage products (initially backup).
The afternoon was taken up by a trip to Data Robotics, creators of the Drobo storage device. This session was probably my most anticipated of the week and I wasn’t let down. It was superb – for a number of reasons. Firstly, CEO Geoff Barrall took us through the product in more depth, answering some of the questions I’ve always had around the way Drobo’s BeyondRAID technology operates. This will become a post in its own right, expect to read it over the coming week.
The Field Day delegates then split into groups for a practical session, followed by a questionnaire, the winners of which would receive a Drobo each. The winners were; Team #1 – me, Rod, Devang and Simon! Now I have two Drobo devices for my data.
The most anticipated part of the Drobo session was the unveiling of two new Drobo products. Details are under embargo until November 23rd, however I can say the new devices will fill out the Drobo range very nicely. Expect a post on this as the embargo date is reached.
Day 2 wrapped up with dinner at a local italian restaurant and signalled the start of delegate’s long trips home.
In summary, the first Gestalt IT Tech Field Day has been a great success. The presenters have been open with their product information, keen to discuss details and helped to provide a lot of insight into their thinking and strategies. Fortunately, no-one fell into the “marketing slideware” trap. Expect to see posts over the coming weeks and days that delve into some of the details we’ve seen over the last two days. As a group we’ve covered a huge number of topics to even attempt to put into a single post.
Congratulations to Stephen Foskett for setting the event up. A great deal of thanks also need to go to Claire Chaplais and Sunshine Mugrabi, without whom the event wouldn’t have happened. One final thought; where and when will Tech Field Day 2 be?
Disclaimer/Disclosure: The sponsors and presenters of the Gestalt IT Tech Field Day event have only paid for my accommodation and travel expenses in San Jose during this trip. I am not employed or paid by any of the sponsors/presenters to express my views. I have received a Drobo storage device as part of the competition held at Data Robotics during the Field Day. I also received a nice Drobo body warmer & T-Shirt and a T-Shirt from 3Par, plus a number of assorted USB memory sticks.
Day 1 of the Gestalt IT Field Day started early at 7am with a trip to VMware and their executive briefing centre where breakfast was provided. As well as food, there was an opportunity to see the “data centre in a rack”, used at VMworld to run all of the demo and presentation materials. Once replete, we were whisked off to another VMware office for presentations from MDS Micro, Xsigo and VMware. At this point the choice of sponsors became clear, as the integration of high density server hardware (MDS Micro), I/O virtualisation (Xsigo) and virtualisation software (VMware) was used to create the “data centre in a rack” we saw earlier. Whilst the combination of the three offerings was compelling, I think it may be some time before highly compact computing offerings like this are widely adopted. The lab test was testament to this; the simplicity with which virtual HBAs and NICs could be created lends itself to virtual device sprawl unless clear best practices and standards are put in place. That said, this type of computing power will be the future for many organisations.
From VMware, we moved on to 3Par and a presentation on the InServ hardware, with a follow up from Symantec. Again the parthership message was clear. 3Par provide thin provisioning with the ability to reclaim unused resources. This is achieved today through detecting patterns of data with no content (binary zeros), but in partnership with Symantec will be integrated into the Veritas File System. The 3Par presenters were excellent and the only disappointment was the lack of time to talk. To be fair, it would have been possible to take the whole day just discussing 3Par alone.
Next stop was the Computer History Museum for the first evening party. This was a chance to catch up more informally with the sponsors. It was also an opportunity for me to talk with The Notorious 3P (otherwise known as Marc Farley). The museum was fascinating, with exhibits ranging from Babbage’s Difference Engine to the first Apple I and a range of computing devices from my youth (the Commodore Pet, Sinclair Spectrum and many many others).
Festivities continued at the bar of the DoubleTree where the heated discussion turned to the subject of thin provisioning with VMware. it’s interesting that thin provisioning still causes serious debate.
All in all, the day was extremely successful, The depth and quality of the content in the presentations was very high; the whole day ran without a hitch, to which the organisers (Stephen, Claire and Sunshine) deserve all the credit.
In the coming weeks I’ll discuss in more detail the content of today’s presentations, including more talk about thin provisioning. In the meantime, here are pictures from the day.
Here are a few pictures from tonight’s dinner; some people turned up later than planned and joined in as they arrived. Congratulations to the restaurant staff who served them so quickly. Also congratulations to those who won prizes from Stephen’s cryptic competition. We’re expecting good things from Devang as he won again with video hardware. Let’s see some videoing!
Finally arrived at the DoubleTree in San Jose. Met up with Devang, Nigel and Simon at the airport and shared a leisurely hour discussing our thoughts as we travelled to the hotel. Thanks to Stephen and Claire for a great limo!
Whilst checking in we bumped into Robin Harris in the foyer. I’ve read Robin’s blog for quite a while and it was great to finally meet him in person.
Now it’s time for ablutions and off to the bar in a vain attempt to keep awake (8 hour lag for us Brits). More later, possibly including some photos from the bar (if they’re printable!).
