The Legacy View

Legacy Provisioning Proces

For the last 10 years or so, the traditional view of  storage management has consisted of a number of Storage Administrators using a GUI, CLIs and/or scripts to process storage requests as they are generated by the business user.  The process is highly manual, with lots of interactions between the requestor, the storage admin delivering the work and other intermediaries to cover things like billing, change control, capacity management and workload scheduling.  This made the overall process pretty people intensive and not surprisingly elongated the delivery time.  Many end users will also have recollections of asking for their specific requirement to be told they can only have something “off the shelf” – i.e. storage to a standard LUN size and with a specific RAID protection.  This was done for obvious reasons; firstly the configuration of large arrays was predicated on pre-planning and a fixed design, usually created at hardware installation time.  Once defined and in use, it couldn’t be changed (or at least couldn’t be changed without significant impact and cost).  Second, it makes sense to reduce requirements into a smaller subset to make the provisioning process easier.  As well as being rigid in configuration, many legacy arrays assume the creation and provisioning of LUNs is an infrequent task.  Many require requests to be packaged and executed in batch and certainly can’t cope easily with concurrent requests.  Although it is possible to automate some provisioning processes using CLIs and scripts, this doesn’t address the real requirements in creating an on-demand model.

The New World

API Provisioning Process

As we scale up to ever large IT deployments and especially within service-based or “cloud” configurations, the idea of having large amounts of human intervention in the provisioning process simply doesn’t work.  Instead, we need to move to a model of “storage on demand” where an external agent – user or orchestration software – can request storage as part of a portal and see the request actioned in real-time or at least within a matter of minutes or hours.  This kind of operation can only be delivered where the hardware has been designed for the purpose.  Where previously storage administrators were involved in every provisioning request, those requests will be actioned within a provisioning framework, defined by the administrator or a storage architect.

Framework

What do we mean by framework?  Well, it’s all about setting a set of parameters around which allocations take place.  This could include:

• LUN Size
• Resiliency/Availability
• Performance
• Security credentials
• Snapshot policy
• Capacity on demand LUN

Designed for API

Of course there’s a big question around whether APIs can be retro-fitted to existing storage.  In classic IT tradition the answer is “it depends”.  Without a doubt no new storage array should be released without native API functionality.  However fitting an API to existing technology will depend on how flexible the existing configuration process is.  It’s possible to create an API wrapper and build automation into a middleware layer.  This is how products such as iWave’s Storage Automator work.  However adding these features to existing storage products could be costly and still be an imperfect solution.

The Storage Architect Take

As new storage platforms evolve, native API support is a must.  The Storage Administrator will simply be required to deploy the infrastructure and plug it into a higher framework from where provisioning will be entirely automated.  Vendors offering this level of functionality will be the most attractive to service providers, looking to make the cost of acquiring and managing storage as cheap as possible.  We’re about to see a paradigm shift in the way in which storage is managed and possibly an end to the storage administrator.

Available Now as Public Beta

• Compute – on-demand server instances.
• Cloud Object Storage – object-based storage using RESTful APIs.
• Content Delivery Network – local distribution of web content.

Still in Private Beta

Cloud Storage

The concept of Cloud Storage is pretty simple.  Services like Dropbox allow you to share a local folder on your PC or Mac and have that data replicated into “the cloud”.  From there it can be accessed by other devices, including smartphones, web browsers and tablets.  The great benefit of cloud storage offerings is that they allow all copies of data to be kept in sync, while retaining a backup copy that can be used if any or all of the local device copies are lost or corrupted.

My Data?

You’d think that because you were uploading your files, that you own them.  Whilst that’s true, what you can’t tell is whether your data is being used by your cloud provider for something else.  As I highlighted when Google’s service was announced, their terms and conditions state the following:

Some of our Services allow you to submit content. You retain ownership of any intellectual property rights that you hold in that content. In short, what belongs to you stays yours.

Great, but read on and the Terms of Service also say:

When you upload or otherwise submit content to our Services, you give Google (and those we work with) a worldwide licence to use, host, store, reproduce, modify, create derivative works (such as those resulting from translations, adaptations or other changes that we make so that your content works better with our Services), communicate, publish, publicly perform, publicly display and distribute such content.

This amount of freedom with my data seems a little generous;  effectively Google (or any company they choose) can redistribute my content, including modifying it, for their purposes.  Now it could be argued that Google are providing this service for free and somehow they need to recover their costs, but as others have done, when you offer a “free” service you rather hope to upsell the customer to a paid offering, either by giving enhanced features or capacity and cover the cost of offering the service to some for free.

Encrypt

So far I’ve chosen not to write any content into the Google Drive while I work out the best way to encrypt my data and still have it accessible across all platforms.  Ultimately if you’re not happy with what Google are doing, then encryption is the way forward.  The trouble with encryption though, is it restricts the use of plugin apps (like Google Docs) and reduces the effectiveness of sharing, when keys have to be shared around those accessing the shared data.

Shared Metadata Local Content

There is another way this problem could be solved and that’s to develop a hybrid solution.  Content can be separated from the metadata and access method, allowing users to retain data on their equipment, using a provider to store metadata and provide the shared access API.  This is what OxygenCloud now offer and VMware will offer with Project Octopus.  These kinds of solutions allow customers to retain control over their data whilst having the benefit of Dropbox-like functionality and eventually (as these service providers hope) to store only their critical information locally, pushing more into the Cloud (and their partner service providers) as they trust security and service levels.

The Storage Architect Take

Google will do well with Drive simply because they have a critical mass of users who like “free” and don’t care about the security and data access issues.  Pushing Drive into corporate territory will require a different approach; charging a fee and offering encryption and/or better terms of service may be one way forward.  However, in the meantime, there are plenty of others out there already eating Google’s lunch.  The hybrid on/offsite model will develop and mature.  The hybrid solution is something I can’t see Google engaging in; they like to own everything, so perhaps Drive will not be as successful as Google would like it to be after all.

Rackspace – All virtual machines get 4 vCPUs, which are based on the 18-month old AMD Opteron 2374HE processor running at 2.2Ghz.  Virtual machines have a bogomips figure of around 5500.

Amazon AWS – Amazon use Intel processors in the EU-West data centre that I ran my test on.  For a large instance, the CPU in question was an Intel Xeon E5507 (released 1Q2010) running at 2.27Ghz.  This processor gets a bogomips figure of 4533, somewhat lower than Rackspace.  This large instance has 2 vCPUs with 2 EC2 units each.

GoGrid – GoGrid use Intel Xeon E5520 processors at 2.27Ghz, with a similar bogomips rating to Amazon of 4533.  This isn’t surprising as they run at the same speed.  GoGrid quote CPU for a base VM as being between 0.5 and 1 core.

I’ve amended my script and produced something a little more formal – see the attached “cloudperf.pl” file.  This runs a number of maths intensive loops, with the option to run multiple threads and multiple iterations.  Performance is measured in MFlops, which is simply a count of the number of loops achieved in the script, in millions.  You can also change the time each loop runs for.  Just type “perl cloudperf.pl” –help for details.  Note:  There is no warranty or guarantees given with this script and you run it at your own risk.

Curiously, I’ve found an anomaly with a couple of virtual servers I’ve built over at GoGrid.  The smaller of the two is running at their US-West location and appears to have a standard CPU setup, however the output from my script shows the following:

Script is running with the following settings:

Number of threads: 1
 Number of iterations per thread: 10
 Seconds per iteration: 1

Iteration     PID           MFlops    Real      User      Sys
 1             22141         0.986    1.000    0.510    0.000
 2             22141         1.942    1.000    1.000    0.000
 3             22141         1.732    1.000    0.900    0.000
 4             22141         1.876    1.000    1.000    0.000
 5             22141         1.935    1.000    1.000    0.000
 6             22141         1.922    1.000    1.000    0.000
 7             22141         1.930    1.000    1.000    0.000
 8             22141         1.734    1.000    0.900    0.000
 9             22141         1.934    1.000    0.990    0.000
 10            22141         1.898    1.000    1.000    0.000

Compare this to an instance running in US-East, which has more cores of the same type.  Remember I’m just running single threaded here:

Script is running with the following settings:

Number of threads: 1
 Number of iterations per thread: 10
 Seconds per iteration: 1

Iteration     PID           MFlops    Real      User      Sys
 1             1101          0.250    1.000    0.190    0.220
 2             1101          0.593    1.000    0.410    0.590
 3             1101          0.593    1.000    0.420    0.580
 4             1101          0.604    1.000    0.450    0.560
 5             1101          0.603    1.000    0.440    0.550
 6             1101          0.599    1.000    0.460    0.540
 7             1101          0.606    1.000    0.410    0.600
 8             1101          0.606    1.000    0.440    0.550
 9             1101          0.604    1.000    0.410    0.590
 10            1101          0.602    1.000    0.450    0.550

Performance is roughly 1/3 that of the other instance for identical CPU types.  The difference is in the User versus System time.  The slower instance is being constrained by the hypervisor (showing more system time), whereas the US-West one isn’t (all User time).  This implies US-West isn’t busy and when that happens, the instance can run at the full rate of the CPU.  Of course I’m taking these measurements during the working day in GMT, whereas many of the servers in US-West could be serving traffic on the US west coast and be quiet at the moment.  I’ll need to repeat this test in a few hours and see the results.  GoGrid don’t specify exactly what performance is guaranteed other than to state that my instance will get between 0.5 and 1 cores – whatever a core is.  I’ll post the “daytime” US-West figures later.

So, I’ve just run the numbers again on my West Coast GoGrid instance:

Script is running with the following settings:

Number of threads: 1
 Number of iterations per thread: 10
 Seconds per iteration: 1

Iteration     PID           MFlops    Real      User      Sys   
 1             23133         0.229    1.000    0.110    0.000    
 2             23133         1.879    1.000    1.000    0.000    
 3             23133         1.923    1.000    1.000    0.000    
 4             23133         1.910    1.000    1.000    0.000    
 5             23133         1.891    1.000    0.990    0.000    
 6             23133         1.729    1.000    0.910    0.000    
 7             23133         1.937    1.000    1.000    0.000    
 8             23133         1.926    1.000    1.000    0.000    
 9             23133         1.910    1.000    1.000    0.000    
 10            23133         1.889    1.000    1.000    0.000    

It’s still running with the same level of performance.  Does this mean nothing is running in this GoGrid data centre?  So there’s  a market here for the ability to determine where virtual machines are running fastest and sell those as the best locations to host in…. until everyone does it.

Download cloudperf.pl

CPU or processor power is described by most vendors in terms of cores, which translate to some abstract definition of physical computing power.  Only Amazon Web Services (AWS) reference physical CPU architecture, with processing assigned EC2 Compute Units (ECUs).  You can find more details here, but summarising, an ECU is approximately the power of a 1.2Ghz 2007 Intel Xeon Processor.  Memory is a more tangible quantity and simply expressed in megabytes or Gigabytes.  Storage references purely disk capacity and has no correlation to actual disk performance.

Being a “storage guy” this lack of an I/O performance metric piqued my interest, as much of my professional career in storage has involved ensuring consistent and high I/O performance.  I thought it would be interesting to look at both processing power and disk I/O performance to see how the different cloud implementations match up.

Measuring Performance

Now I could install some software tool to execute the performance tests, but it’s more interesting to think about the underlying processes that are occuring on a virtual machine, so I’ve created a couple of PERL scripts to do the analysis.  For the CPU measurement, I’ve simply created a script that loops for a fixed number of seconds, performing maths calculations and counting the number of loops that get executed in that fixed interval; in this case one second.  A single measurement isn’t an entirely accurate measure of performance so I repeat the process at one second intervals, obtaining a series of figures that can be averaged out.

For storage I/O my PERL scripts creates a 100MB file, writing a series of random 4K data blocks.  With both scripts I measure elapsed time and the user & system CPU time taken.  If the PERL script is being executed consistently, then CPU time for each metric should be similar across all cloud environments although the elapsed time will vary by the percentage of resources being allocated.

The Results

I ran tests against the following platforms: Amazon AWS, Rackspace and GoGrid, all of which were the US-based service.  I also tried to choose a consistent platform, standardising on CentOS or RHEL (which should be identical).  Unfortunately there is no standard version of these operating systems available on each platform, so some tests are based on version 5.6, some on 6.x.

• AWS#1: RHEL 6.1, 2ECU (burst only), 613MB, I/O performance low
• AWS#2: RHEL 6.1, 2ECU, 7.5GB, I/O performance high
• AWS#3: CentOS 5.6, 2ECU, 7.5GB, I/O performance high
• Rackspace: CentOS 5.6, 4 virtual cores, 256MB, 10GB
• GoGrid: CentOS 6.0, 0.5 CPU Core, 512MB, 25GB disk

What’s interesting is most of the CPU performance figures came out at a similar level, except for the AWS micro-instance.  This gets more power, but after about 10 seconds of continuous use starts to get throttled.  All of the instances are different in their definitions of computing resource but effectively translate to the same amount of CPU speed (remember the script runs single threaded).

For the storage, most instances ranged between 1 & 2 seconds per 100MB file.  However, the two AWS instances using Elastic Block Store (permanent data store, retained even if the instance is destroyed) have significantly worse performance, with the micro-instance being particularly bad.  One curious anomaly is that performance seemed to improve for the micro-instance in line with the way CPU was constricted.  Although the timings were rounded to the nearest second, taking the average of all the observations, the three solutions using instance storage came out at a remarkably similar time, although AWS was slightly faster (1.4s per 100MB compared to 1.7).

So what’s the point of performing these measurements?  Well firstly, it provides an additional way to do better like for like comparisons of the different offerings.  Obviously I/O performance is not part of the server profile but can vary dramatically, depending on the instance type chosen.  If, over time, servers are migrated to new technology, the relative performance level can be evaluated ensure servers remain correctly sized.

I’ll be extending the scripts to do more complex performance testing and see if I/O varies with differing block sizes and how multi-threaded CPU tasks are handled.  Plus, there’s the whole comparison of Linux versus Windows to contend with.

First, there’s undoubtedly going to be security concerns that all of your browsed content is going through Amazon’s servers and will be tracked, filtered and analysed.  This is done to “improve the browsing experience” but ultimately gives Amazon access to a wealth of information on your personal browsing habits.  The information would make great content for a retailer to trawl through but on a slightly more sinister angle could expose a lot of your personal information to hackers.  One presumes secure HTTPS connections over SSL will still be secure, but everything else will be up for grabs.

My other thought arises from a comment made on the Silk website (see here).  Amazon are making a virtue of the fact that many websites are on EC2 already today and so latency to their analysis servers will be very low.  Does this mean that Silk offers Amazon an opportunity to cross-sell their EC2 platform to those organisations looking to deliver web content on the Kindle Fire?  It seems like a clever business model.  Host your web servers with us and we’ll get you closer to the user.

I’m not sure my browsing experience is so slow as to justify needing server-based acceleration at this time.  Perhaps it will in the future, who knows.  What I do know is I’ll be making sure I’m typing https:// rather than http:// wherever possible.

It sounds like a great thing to do, however maths is against him. With 26 letters, doubled for upper and lower case, plus maybe a dozen punctuation symbols and the space, each letter multiplies the possible combinations at least 60-fold. I have no idea how many letters are in Shakespeare’s entire works, but it’s too many to make this task practical.

Whilst this is a good storyline, it demonstrates the power of splitting and distributing computing tasks, something that cloud computing is good at, even if it isn’t cost effective.

In this instance, brute force won’t solve the puzzle. Instead a compromise has been made to exclude any 9-character segment that matches part of the Shakespeare text. This demonstrates that distributed “infinite” resources alone are not enough to solve a problem and we’ll be relying intelligent programming and clever algorithms to make distributed cloud computing really work.

The Chromebook itself is essentially an underpowered laptop – you could also think of it as an oversized notebook computer.  Underpowered doesn’t necessarily mean competitively priced as the device will cost around £400 in the UK, putting it on a par with a reasonably well specified laptop or tablet.  Of course to do anything you need a network connection of some sort – Wifi or 3G.  This is stumbling block number 1.

All applications are accessed online, which means if you have no network connection, you have no usable device.  Google (via AT&T in the US) are offering plans that start at 100MB per month for mobile data.  This is a tiny figure based on the fact that all your email, photos, documents and anything else you have would need to be both uploaded and downloaded through a wireless connection.  Google must be hoping that wireless connectivity becomes ubiquitous.  I can’t see how this is at all likely within the next 20 years.  Firstly today’s 3G technology isn’t capable of delivering a constant reliable connection.  Buildings mask or break the signal and base stations can’t cope with large numbers of users; most countries still don’t offer Wifi on aeroplanes.  Second, economically there’s no business case for rolling out 4G networks to every single pocket of the world.  Look at any map of 3G coverage today and it’s concentrated around population areas and sometimes major transport links.  Go “off piste” and you have no signal. This was done for a reason.  Base stations cost money to deploy and there’s an economic cost/benefit model against deploying them in low population areas.

The next major issue is control.  All applications have to be accessed via Google; the first requirement of using the Chromebook is to log into your Google account.  All applications are then delivered via Google’s App Store.  Many people will say this is no different to the way Apple control the iPhone, iPad and iPod devices.  That is true but there are other devices coming to market that challenge this approach.  The control issue is much wider than simply the management of an App store.  Google store your data on their platform and like all vendors, don’t make things easy when you want to move your data elsewhere.  This is a major issue with the way cloud services are offered today – data inertia.  We have had the issue in Enterprise computing for many years; proprietary database formats, storage arrays that don’t replicate to the competitors products.  In fact the situation is getting worse, as vendors are starting to bring out converged infrastructure that retains the grip over customer data.  Having a laptop that offers local data is still a necessary security blanket.

Then there are the alternatives.  Google are pitching the Chromebook at schools and businesses, making a virtue of the ability to log into any device to access data.  Well we can do that already with Virtual Desktop implementations.  Thin clients are much cheaper than the Chromebook in terms of pure hardware cost.  They also have the same flexible benefit of not storing user data.  The tipping point comes in determining the cost of deploying and managing VDI infrastructure versus using a solution such as Google’s.  As networks get faster and more reliable, why not just use a hosted VDI solution?

The Architect Angle

Google have to do something to meet the challenge from Apple, Amazon (and to a certain extent Microsoft) and others in their core business – delivering data and applications via the web.  I think the “always on” assumption is at this stage flawed and there are plenty of other better solutions already available out there.  Consumers will balk at paying £400 for a device which has no other purpose than to connect to Google.  At that price it is better to purchase a TouchPad or iPad and have access to multiple server providers.  Google have a long uphill struggle ahead of them to get Chromebook into the dominant position Apple enjoy today.