Lightning AKA VFCache

VFCache (Very Fast Cache) is the final product name for the project that was called Lightning.  It turns out that this offering is nothing more than a PCIe SSD card for servers (not all servers mind you, but at this stage quite a few).  The initial offering combines with software to act as a very fast read cache to the host.  In Windows terms, this is implemented as a filter driver that sits above the STORPORT driver in the I/O stack, with similar implementations on Open Systems platforms.  The software component of VFCache tracks I/O and caches reads in order to speed up future I/O requests without needing to go to external disk.  Writes to disk are not cached by VFCache and EMC tried to make a virtue of the fact their product acts as a “write-through” cache, meaning I/O writes have to be committed to physical disk before the cache acknowledges them to the host.  Rather than being a benefit, write through mode in this instance is more likely to make the cache less effective by polluting the cache with writes that can’t be released until confirmed externally.  When there’s a difference in I/O if microseconds versus milliseconds, then this difference really matters.  However, I don’t think this is a design flaw, merely a placeholder for the future, as I’ll discuss later.

Disappointingly for EMC, VFCache 1.0 really is a 1.0 version in terms of feature support.  Within VMware ESXi for instance, the card installs with a device driver that only allows the cache benefits to be used when the filter driver is deployed into each ESXi guest, so it’s not simply a case of insert the card and off you go.  Moreover, the VFCache appears as a DAS device within VMware and so can’t be used in conjunction with vMotion.  For many organisations this is a huge omission as there’s a big correlation between high performance and high availability; the lack of vMotion isn’t acceptable.

The Competition

We can’t go any further on the VFCache discussion without mentioning the competition and in one of the presented slides, EMC paid homage to the market leaders, Fusion-IO.  Their ioCache product already accelerates VMware ESXi and Windows 2008 environments, using a similar hypervisor plugin approach.  ioCache already offers double the capacity of VFCache and it’s likely Fusion-IO have larger capacity cards in the pipeline as they already offer a range of SLC and MLC flash devices.

Thunder Follows Lightning

Surely Hitachi must be enjoying the irony of EMC choosing product code names based on already defunct HDS hardware (Thunder and Lightning were the mid-range and Enterprise products respectively that preceeded AMS and USP).  The next product announcement moves the flash-in-server story forward and explains how this technology is limited in terms of availability.  The move to centralised SAN environments was done precisely to fix the issues that occur with server-side SSD today.  Data is locked into the server, is difficult to expand (requiring downtime and physical intervention) and is isolated from access if a physical failure of the server should occur.  So, step up Project Thunder, EMC’s purpose build all-flash array.  This device allows multiple servers to share data across what EMC are calling the “server-area network”.  What they mean is a physically local, high-speed interconnect (such as Infiniband or Rapid-IO) between the server and a fast all-flash storage array.

The interconnect technology between devices already exists today (as already mentioned) but placing it into the server and using this for shared storage presents more of a challenge.  Where Lightning was a simple Filter Drive, Thunder will require deeper integration in order to manage consistency across all connected servers.  This isn’t something that comes new to EMC – think of how VMAX nodes interconnect and you have the model already there.  However, the implementation may require closer co-operation with server vendors than EMC can achieve, those same companies they are already in competition with for storage – IBM, HP and Dell.  This could mean Thunder becomes a VCE only product or is severely restricted when deployed in other manufacturers hardware.  We will have to wait and see. (Side Note: This also means that other all-SSD array manufacturers could become more attractive to HP, IBM & Dell as acquisition targets – check out here and here).

The Competition

Let’s not forget that dedicated all-flash arrays are already out there.  Recently I’ve discussed Pure Storage and SolidFire and there’s also Violin Memory, who have been going at this market for quite some time.  They already have the SSD array technology to a mature level including support for Infiniband; all that’s needed is a software driver to bring clustering to their products.

Futures

What can we expect going forward?  There are lots of gaps the product releases we’ve seen today.  Lack of vMotion support and no write-back cache to name only two.  The question we should be asking is what could be delivered in the future.  EMC have access to every piece of the I/O stack, from the hypervisor, the multi-path driver, through to the array.  Using PowerPath, EMC can develop more intelligent algorithms that choose whether to cache I/O locally in the server/hypervisor, destage to the array, leverage pre-fetching from disk and other clever ways to squeeze the best level of performance out of the hardware stack.

Conclusions

EMC have ratified both the PCIe SSD and dedicated flash array markets with their announcements of Lightning and Thunder.  At this stage they are bringing only “me too” products to the market, with other vendors out there having already delivered more advanced technology than being announced today.  However EMC have two big advantages; (a) they are a huge organisation, with access to the majority of customers in the market and a great marketing team.  They have the ability to place their products into customer environments and use price as the main differentiator (b) they have a huge R&D budget and never stand still on product development.  Today’s 1.0 releases will be superseded within months and address some of the shortcomings we can see today.  The future battle will not be over the hardware, but the software that integrates I/O in the server to I/O on the array, delivering the benefits of local flash with the safety of external storage.  The eventual winner will be the vendor who gets that software and hardware integration right.

Related Articles

You may be interested in the following related articles from this and other sites.

• Fusion-IO Shares Tumble as New Entrants Prepare to Enter The Market
• Emulex – Evolution of the HBA
• Solid State Arrays: SolidFire
• Solid State Arrays: Pure Storage Inc
• Who Will Be The First Solid State Array Vendor To Be Acquired?
• HP & Violin?
• Enterprise Computing: Violin Memory Inc Release New All-SSD Array
• EMC VFCache (aka “Project Lightning”) Is One Small Step, But an Important One (Stephen Foskett)
• Cache Splash (Storagebod)
• VFCache Means Very Fast Cache Indeed (Chuck Hollis)
• VFCache: Hello World! (…and covers come off Project Thunder) (Chad Sakac)
• My take on EMC’s project lightning (Enrico Signoretti)

This isn’t the first reality check of its kind; think about the untouchable STEC in the SSD market who’s shares once traded at almost $45 and are now just under $10.  Their demise was competition and the reliance of a single customer, EMC, who decided one day they had enough supply, thanks, as the SSD market wasn’t evolving as quickly as they had planned.  It seems the spectre of EMC is about to strike again, as Project Lightning, EMC’s PCIe SSD device draws close to release on 6th February this year.

Is it all bad for Fusion-IO?  Probably not, this is nothing more than a correction of an overpriced stock and doesn’t mean the company is in trouble.  Fusion-IO continues to innovate as we heard recently with the announcement of their Auto Commit Memory feature, which extends memory onto the PCIe bus using kernel-based code changes.  The idea is an interesting one; extend what in the old days would have been termed “real memory” using the next fastest level of storage, PCIe SSD.

Technically, I don’t think this change is that difficult to achieve.  Modern operating systems already have interrupt-based memory management, using virtual memory pages that are swapped into real memory as required.  Although I’ve not seen the details, Fusion-IO could have either used their PCI-SSD device as very fast cache paging device, rather than disk, or extended the available physical page slots the host can see.  It seems that the former rather than the latter is more likely, as the literature released discusses “flushing in-flight data”, something that’s not relevant with volatile physical memory slots.

Back to the competition; EMC may be coming into this market, however their product will be a 1.0 release.  Nevertheless, EMC already own the storage array and hypervisor, so it’s not a difficult leap to imagine that the first target for their Lightning technology will be in accelerating VMware ESXi deployments.  This means they would already be enhancing the kernel of that platform, giving them a great headstart on others in the marketplace.  As we see move towards the majority of server deployments being virtualised, the hypervisor is the place for this technology to sit.  Will VMware be willing to provide FusionIO and others the same kernel-level access as EMC must surely be getting?  Who is working with Microsoft to deliver this technology into Hyper-V?

In many respects, PCIe SSD devices are only at the very early stages of development.  In the coming months and years we are going to see some very interesting times indeed.

However, the VNX is still a CLARiiON at heart (with all the legacy baggage that entails), so one wonders what additional capabilites the VNX5500-F has to cope with this huge I/O workload and of course, to manage the finite lifetime of SSD devices.  More important is the ability to cope with the fickle performance of I/O spikes that are associated with SSD garbage collection.  There’s no mention of how (or even if) EMC have added technology to cater for these issues.  Bearing in mind what an all-flash array will cost, then 100% guaranteed low latency of every I/O will be expected.

The new market startups (Violin, Pure, SolidFire & others) will have to compete against EMC’s marketing machine but to be fair this is a technology that already offers a wide range of features, including connectivity via all the common protocols in use today.  EMC will be able to sell simply on feature, functionality and support.

The VNX5500-F may seem like putting a Rolls Royce engine in a mini compared to the competition, who are more like thoroughbred Ferraris and Lamborghinis, however as usual, cost will be the ultimate decider.  EMC don’t quote price, but simply indicate that cost per TPM is vastly reduced.  I’d like to see some real world list costs from EMC (which won’t happen) plus some statements on how this dedicated VNX deals with the particular issues of SSD drives.  If you are considering an all flash array, then these questions need to be on your list.

• Unified NAS & modular SAN platform -VNX
• A “baby” VNX, the VNXe
• Upgrades on performance & features to VMAX (some of which have been available for some time)
• Performance upgrades to Data Domain products
• The addition of Isilon to the EMC family

Full details of all the announcements are at http://www.emc.com/microsites/record-breaking-event/index.htm

VNXe

The new VNXe product gained a lot of attention and EMC did remarkably well at laying on the cheesy superlatives, with a ten-year old child actor replacing a failed drive onstage, managing the array through his iPad.  Undoutably VNXe is aimed at the low end market that Dell (with Equallogic), HP (Lefthand), Netapp (FAS2xxx) already occupy.  This has been an area EMC have had no real product offerings but is becoming increasingly competitive, with IBM releasing their new StorWize array into this sector last year.  EMC had to do something and VNXe isn’t a revolutionary release, merely a catchup.  For a start, their definitions of “unified” protocols covers only CIFS/NFS and iSCSI; no fibre channel support at this time.

VNX

The main VNX family merges the previous CLARiiON, Centerra and Celerra products into a single platform.  These arrays can be said to be “unified”, offering NFS/CIFS, MPFS, pNFS, FC, ISCSI and FCoE protocols in all except the low end VNX5100.  These arrays also start to use SAS drives and a full SAS backend, something other vendors (like Hitachi) have been doing for some time.  There’s no doubt that the new VNX arrays are pitched firmly against competition from the likes of Netapp, who have taken full advantage of the serendipitous nature of their NAS architecture in virtualised environments. However that said, a single mid-range product line makes complete sense from EMC’s manufacturing and support perspective as well as for customer understanding of their product offerings as new products are continually being added to the range (think Greenplum, Isilon and Data Domain as recent acquisitions).  Clearly it remains to be seen whether the performance claims of the new platform are met in the real world.

Speaking of claims, EMC made many in the course of their presentation; VMAX Fast Savings in power could power 24,400 homes; EMC shipped 10x more flash than their nearest competitor; 2x performance throughput improvements on VMAX with only a code upgrade.  I could go on.  Unfortunately these figures are meaningless and misleading without independent verification.  EMC still refuses to submit their products for generally accepted (SPC) benchmark verification and the figures on the presentation today mainly credited EMC Internal Testing or were too obscured to be readable.

Overall I think today’s product announcements consolidate EMC’s product strategy, provide incremental improvements but in general don’t offer anything that isn’t already available in the marketplace.  The performance and efficiency claims should be challenged by customers at every opportunity as only customer pressure will make EMC change their policy.

One final note; the “breaks records” strapline was accompanied by two stunts – onstage EMC crammed 26 people into a Mini and “live” from Miami, an Evel Knievel wannabe jumped 40 empty VMAX cabinets in a new record for jumping virtual storage on a motorcycle.  I’m curious to know, who previously held that array jumping record and how long did it stand for…..

Dell saw the writing on the wall some time ago and moved out to Poland where operations were clearly deemed to be cheaper.  But what about EMC?  A large amount of their manufacturing takes place in Cork; of course a lot of it is simply assembly and testing and that could be moved any cheaper alternative.  EMC are moving further towards commodity hardware based on Intel technology, making the whole assembly and testing process easier and cheaper.  For instance, I wonder if the same level of testing takes place with VMAX and CLARiiON as does with Symmetrix arrays?  If not, could that assembly work be moved to another location where the complex and expensive testing facilities are no longer required?

Then there’s the question of stability.  Ireland looks to be heading for a general election, so politics as well as economics are unstable.  This could be another deciding factor on leaving for more favourable shores.

There is a lot of other investment by EMC in the Cork facility; they run helpdesk facilities from there; there’s a large customer briefing and testing centre, so perhaps the Cork investment is too much to give up.  However nothing is ever guaranteed.

I wonder if Storagezilla can speak Polish?

Using the analogy of bespoke car production and fleet management isn’t the best comparison to be made.  Firstly as we all know, Henry Ford revolutionised the production line in order to reduce costs and bring more affordable transportation to the masses; it improved the production process but removed choice from the customer – “Any Colour as Long as It’s Black” is the oft overused expression we think of.

As this analogy extends to fleet car management Chuck implies having choice is a bad thing:

“You’d be spending a lot of time and effort with your car assembly people.  Maybe lots of debates as to who has the best engine, transmission, seats, etc.  I can easily imagine a flock of vendors descending on you with glossy powerpoints and blueprints trying to convince you that their components are better choice for your next kit car.” – Chuck Hollis

Again, this premise is based on false assumptions.  Fleet car owners are no different than any other group of people who have to source a product to meet a need.  If the requirements are set out clearly at the beginning of the process of product (car or technology) selection, then glossy brochures are irrelevant; the vendors have to meet your stringent set of requirements, simple as that.  For the user of the fleet car who isn’t paying for their vehicle, they would freely admit that the car probably wouldn’t be their first choice.  Many companies now allow employees to opt out of the managed scheme, instead providing an allowance to self fund a vehicle. Why?  Because fleet management costs money.

However if we think deeper about the concept of fleet cars, bespoke cars and converged infrastructure, then Chuck’s analogy completely breaks down.  If a fleet car manager chooses a particular vendor and that proves to be a bad choice, then after the lease time has expired, the vehicles can be handed back, with no impact, allowing the fleet manager to choose another vendor.  In fact, the fleet car manager can switch or maintain multiple vendors with little or no impact.  This is because driving a car is a well understood process; if you drive one car, you can drive another and making that “technology switch” has no impact.  However with converged infrastructure the change is not so simple; data has to be moved, processes changed, staff trained on new technology, scripts rewritten; switching between infrastructure vendors becomes a major exercise.

Vendor lock-in is what converged infrastructure vendors are selling, just like IBM did when they owned the mainframe business until the late 80s/early 90s.  If technology from a single vendor works for you, then that’s great, but don’t be fooled into thinking vendors are offering converged infrastructure for altruistic reasons.

Rather than be locked in to a specific vendor, many customers could reach most of their goals by standardisation.  To use the car analogy, cars manufacturers standardised on their products – four wheels, indicator stalks on the driving column, brake, accelerator and clutch pedals in the same place and so on.  Standardisation makes it easy for anyone with a driving licence to drive a car.  Differentiation then comes in the quality/price/features the vendor can then offer.

This also occurs in IT.  You can standardise your product set and make them offer the same features.  You can choose storage products that all offer replication, snapshots and so on.  You can make them integrate into a converged infrastructure without buying the vendor’s stack.  Buying an integrated infrastructure from a vendor isn’t necessarily the best or worse option to take – it’s just another option and Chuck’s implication that vBlock is the only way, well, it’s just plain wrong.

In some respects this move makes sense for EMC, if Netapp are seen as their major competitor in this market.  However as we know, performing this kind of code merge isn’t simple and Netapp themselves have had problems achieving code delivery dates (posts here, here and here).  Perhaps these problems are typical of hardware vendors that are looking to squeeze more out of their increasingly legacy looking hardware.

Of course the biggest confirmation to this theory comes from moves by HP.  They have purchased many new niche storage vendors over the last couple of years, culminating with 3Par only last month.  This is a clear indication that their storage portfolio needed strengthening (and in places replacing) and they’ve acquired new technology to achieve this.

I doubt sales for EMC and Netapp are about to drop off a cliff any time soon, but I would suggest that continuing to lever new features into old architectures is a long term recipe for disaster.  Eventually EMC & Netapp will need a new approach.  Before everyone leaps on my comments, I’ll just remind you all that EMC promised discussed FASTv2 at EMC World in 2009. It has yet to be released as GA code on Symmetrix.  That’s 18 months people.  The world has moved on and FAST is not the only block-level tiering product in town, making it an also-ran rather than an innovation.  Maybe that’s the issue with legacy hardware.

Please Note:  Due to the high level of Spam comments I am receiving (even with captcha-style technology enabled), all commenters will be required to register.  Hopefully I won’t have to force this restriction for too long.

Update: EMC PR have contacted me and asked me to clarify some errors in this post

I have been asked to clarify by EMC PR that FAST wasn’t announced in April 2009 at EMC World but was actually announced in December 2009 as part of this press release.  This is not entirely true.  In fact, FAST was mentioned in the April 2009 release on V-MAX, which can be found here:

http://www.emc.com/about/news/press/2009/20090414-01.htm

The EMC bloggers also commented about it (as did I).  Barry Burke’s post is probably the most detailed:

http://thestorageanarchist.typepad.com/weblog/2009/04/1059-fully-automated-storage-tiering-fast.html

My post is here:

http://www.thestoragearchitect.com/2009/04/14/enterprise-computing-emc-announced-next-generation-v-max-architecture/

So, irrespective of the “official” release date, EMC were selling the FAST story from April 2009 onwards.  As for FASTv2, take a look at this article from Gestalt IT, dated May 2009:

http://gestaltit.com/featured/top/gestalt/emc-symmetrix-vmax-fast-virtual/

FASTv2 was being discussed at this point.  This wasn’t an “official” EMC announcement, but FASTv2 was being discussed at EMC World 2009.

Finally, I’ve been asked to clarify the fact that FASTv2 was GA as from August 24th. This is the press release I’ve been directed to:

http://www.emc.com/about/news/press/2010/20100824-01.htm

It isn’t very clear what the new features of FAST are from this post, and strangely, neither Barry Burke or Chuck Hollis chose to blog about it.  I believe the release only covered the mid-range products, as this Register article discusses:

http://www.theregister.co.uk/2010/08/24/emc_fast_2_flare_30/

So, agreed, FASTv2 is available for CLARiiON & Celerra, but not for Symmetrix.

I’ve tweaked the wording in the post to reflect the specific mid-range reference.  I’ll leave it to your own judgement as to whether you’d rather go with the “official” release dates or the “unofficial” release dates.