Rackspace – All virtual machines get 4 vCPUs, which are based on the 18-month old AMD Opteron 2374HE processor running at 2.2Ghz.  Virtual machines have a bogomips figure of around 5500.

Amazon AWS – Amazon use Intel processors in the EU-West data centre that I ran my test on.  For a large instance, the CPU in question was an Intel Xeon E5507 (released 1Q2010) running at 2.27Ghz.  This processor gets a bogomips figure of 4533, somewhat lower than Rackspace.  This large instance has 2 vCPUs with 2 EC2 units each.

GoGrid – GoGrid use Intel Xeon E5520 processors at 2.27Ghz, with a similar bogomips rating to Amazon of 4533.  This isn’t surprising as they run at the same speed.  GoGrid quote CPU for a base VM as being between 0.5 and 1 core.

I’ve amended my script and produced something a little more formal – see the attached “cloudperf.pl” file.  This runs a number of maths intensive loops, with the option to run multiple threads and multiple iterations.  Performance is measured in MFlops, which is simply a count of the number of loops achieved in the script, in millions.  You can also change the time each loop runs for.  Just type “perl cloudperf.pl” –help for details.  Note:  There is no warranty or guarantees given with this script and you run it at your own risk.

Curiously, I’ve found an anomaly with a couple of virtual servers I’ve built over at GoGrid.  The smaller of the two is running at their US-West location and appears to have a standard CPU setup, however the output from my script shows the following:

Script is running with the following settings:

Number of threads: 1
 Number of iterations per thread: 10
 Seconds per iteration: 1

Iteration     PID           MFlops    Real      User      Sys
 1             22141         0.986    1.000    0.510    0.000
 2             22141         1.942    1.000    1.000    0.000
 3             22141         1.732    1.000    0.900    0.000
 4             22141         1.876    1.000    1.000    0.000
 5             22141         1.935    1.000    1.000    0.000
 6             22141         1.922    1.000    1.000    0.000
 7             22141         1.930    1.000    1.000    0.000
 8             22141         1.734    1.000    0.900    0.000
 9             22141         1.934    1.000    0.990    0.000
 10            22141         1.898    1.000    1.000    0.000

Compare this to an instance running in US-East, which has more cores of the same type.  Remember I’m just running single threaded here:

Script is running with the following settings:

Number of threads: 1
 Number of iterations per thread: 10
 Seconds per iteration: 1

Iteration     PID           MFlops    Real      User      Sys
 1             1101          0.250    1.000    0.190    0.220
 2             1101          0.593    1.000    0.410    0.590
 3             1101          0.593    1.000    0.420    0.580
 4             1101          0.604    1.000    0.450    0.560
 5             1101          0.603    1.000    0.440    0.550
 6             1101          0.599    1.000    0.460    0.540
 7             1101          0.606    1.000    0.410    0.600
 8             1101          0.606    1.000    0.440    0.550
 9             1101          0.604    1.000    0.410    0.590
 10            1101          0.602    1.000    0.450    0.550

Performance is roughly 1/3 that of the other instance for identical CPU types.  The difference is in the User versus System time.  The slower instance is being constrained by the hypervisor (showing more system time), whereas the US-West one isn’t (all User time).  This implies US-West isn’t busy and when that happens, the instance can run at the full rate of the CPU.  Of course I’m taking these measurements during the working day in GMT, whereas many of the servers in US-West could be serving traffic on the US west coast and be quiet at the moment.  I’ll need to repeat this test in a few hours and see the results.  GoGrid don’t specify exactly what performance is guaranteed other than to state that my instance will get between 0.5 and 1 cores – whatever a core is.  I’ll post the “daytime” US-West figures later.

So, I’ve just run the numbers again on my West Coast GoGrid instance:

Script is running with the following settings:

Number of threads: 1
 Number of iterations per thread: 10
 Seconds per iteration: 1

Iteration     PID           MFlops    Real      User      Sys   
 1             23133         0.229    1.000    0.110    0.000    
 2             23133         1.879    1.000    1.000    0.000    
 3             23133         1.923    1.000    1.000    0.000    
 4             23133         1.910    1.000    1.000    0.000    
 5             23133         1.891    1.000    0.990    0.000    
 6             23133         1.729    1.000    0.910    0.000    
 7             23133         1.937    1.000    1.000    0.000    
 8             23133         1.926    1.000    1.000    0.000    
 9             23133         1.910    1.000    1.000    0.000    
 10            23133         1.889    1.000    1.000    0.000    

It’s still running with the same level of performance.  Does this mean nothing is running in this GoGrid data centre?  So there’s  a market here for the ability to determine where virtual machines are running fastest and sell those as the best locations to host in…. until everyone does it.

Download cloudperf.pl

CPU or processor power is described by most vendors in terms of cores, which translate to some abstract definition of physical computing power.  Only Amazon Web Services (AWS) reference physical CPU architecture, with processing assigned EC2 Compute Units (ECUs).  You can find more details here, but summarising, an ECU is approximately the power of a 1.2Ghz 2007 Intel Xeon Processor.  Memory is a more tangible quantity and simply expressed in megabytes or Gigabytes.  Storage references purely disk capacity and has no correlation to actual disk performance.

Being a “storage guy” this lack of an I/O performance metric piqued my interest, as much of my professional career in storage has involved ensuring consistent and high I/O performance.  I thought it would be interesting to look at both processing power and disk I/O performance to see how the different cloud implementations match up.

Measuring Performance

Now I could install some software tool to execute the performance tests, but it’s more interesting to think about the underlying processes that are occuring on a virtual machine, so I’ve created a couple of PERL scripts to do the analysis.  For the CPU measurement, I’ve simply created a script that loops for a fixed number of seconds, performing maths calculations and counting the number of loops that get executed in that fixed interval; in this case one second.  A single measurement isn’t an entirely accurate measure of performance so I repeat the process at one second intervals, obtaining a series of figures that can be averaged out.

For storage I/O my PERL scripts creates a 100MB file, writing a series of random 4K data blocks.  With both scripts I measure elapsed time and the user & system CPU time taken.  If the PERL script is being executed consistently, then CPU time for each metric should be similar across all cloud environments although the elapsed time will vary by the percentage of resources being allocated.

The Results

I ran tests against the following platforms: Amazon AWS, Rackspace and GoGrid, all of which were the US-based service.  I also tried to choose a consistent platform, standardising on CentOS or RHEL (which should be identical).  Unfortunately there is no standard version of these operating systems available on each platform, so some tests are based on version 5.6, some on 6.x.

• AWS#1: RHEL 6.1, 2ECU (burst only), 613MB, I/O performance low
• AWS#2: RHEL 6.1, 2ECU, 7.5GB, I/O performance high
• AWS#3: CentOS 5.6, 2ECU, 7.5GB, I/O performance high
• Rackspace: CentOS 5.6, 4 virtual cores, 256MB, 10GB
• GoGrid: CentOS 6.0, 0.5 CPU Core, 512MB, 25GB disk

What’s interesting is most of the CPU performance figures came out at a similar level, except for the AWS micro-instance.  This gets more power, but after about 10 seconds of continuous use starts to get throttled.  All of the instances are different in their definitions of computing resource but effectively translate to the same amount of CPU speed (remember the script runs single threaded).

For the storage, most instances ranged between 1 & 2 seconds per 100MB file.  However, the two AWS instances using Elastic Block Store (permanent data store, retained even if the instance is destroyed) have significantly worse performance, with the micro-instance being particularly bad.  One curious anomaly is that performance seemed to improve for the micro-instance in line with the way CPU was constricted.  Although the timings were rounded to the nearest second, taking the average of all the observations, the three solutions using instance storage came out at a remarkably similar time, although AWS was slightly faster (1.4s per 100MB compared to 1.7).

So what’s the point of performing these measurements?  Well firstly, it provides an additional way to do better like for like comparisons of the different offerings.  Obviously I/O performance is not part of the server profile but can vary dramatically, depending on the instance type chosen.  If, over time, servers are migrated to new technology, the relative performance level can be evaluated ensure servers remain correctly sized.

I’ll be extending the scripts to do more complex performance testing and see if I/O varies with differing block sizes and how multi-threaded CPU tasks are handled.  Plus, there’s the whole comparison of Linux versus Windows to contend with.

First, there’s undoubtedly going to be security concerns that all of your browsed content is going through Amazon’s servers and will be tracked, filtered and analysed.  This is done to “improve the browsing experience” but ultimately gives Amazon access to a wealth of information on your personal browsing habits.  The information would make great content for a retailer to trawl through but on a slightly more sinister angle could expose a lot of your personal information to hackers.  One presumes secure HTTPS connections over SSL will still be secure, but everything else will be up for grabs.

My other thought arises from a comment made on the Silk website (see here).  Amazon are making a virtue of the fact that many websites are on EC2 already today and so latency to their analysis servers will be very low.  Does this mean that Silk offers Amazon an opportunity to cross-sell their EC2 platform to those organisations looking to deliver web content on the Kindle Fire?  It seems like a clever business model.  Host your web servers with us and we’ll get you closer to the user.

I’m not sure my browsing experience is so slow as to justify needing server-based acceleration at this time.  Perhaps it will in the future, who knows.  What I do know is I’ll be making sure I’m typing https:// rather than http:// wherever possible.

It sounds like a great thing to do, however maths is against him. With 26 letters, doubled for upper and lower case, plus maybe a dozen punctuation symbols and the space, each letter multiplies the possible combinations at least 60-fold. I have no idea how many letters are in Shakespeare’s entire works, but it’s too many to make this task practical.

Whilst this is a good storyline, it demonstrates the power of splitting and distributing computing tasks, something that cloud computing is good at, even if it isn’t cost effective.

In this instance, brute force won’t solve the puzzle. Instead a compromise has been made to exclude any 9-character segment that matches part of the Shakespeare text. This demonstrates that distributed “infinite” resources alone are not enough to solve a problem and we’ll be relying intelligent programming and clever algorithms to make distributed cloud computing really work.

The Chromebook itself is essentially an underpowered laptop – you could also think of it as an oversized notebook computer.  Underpowered doesn’t necessarily mean competitively priced as the device will cost around £400 in the UK, putting it on a par with a reasonably well specified laptop or tablet.  Of course to do anything you need a network connection of some sort – Wifi or 3G.  This is stumbling block number 1.

All applications are accessed online, which means if you have no network connection, you have no usable device.  Google (via AT&T in the US) are offering plans that start at 100MB per month for mobile data.  This is a tiny figure based on the fact that all your email, photos, documents and anything else you have would need to be both uploaded and downloaded through a wireless connection.  Google must be hoping that wireless connectivity becomes ubiquitous.  I can’t see how this is at all likely within the next 20 years.  Firstly today’s 3G technology isn’t capable of delivering a constant reliable connection.  Buildings mask or break the signal and base stations can’t cope with large numbers of users; most countries still don’t offer Wifi on aeroplanes.  Second, economically there’s no business case for rolling out 4G networks to every single pocket of the world.  Look at any map of 3G coverage today and it’s concentrated around population areas and sometimes major transport links.  Go “off piste” and you have no signal. This was done for a reason.  Base stations cost money to deploy and there’s an economic cost/benefit model against deploying them in low population areas.

The next major issue is control.  All applications have to be accessed via Google; the first requirement of using the Chromebook is to log into your Google account.  All applications are then delivered via Google’s App Store.  Many people will say this is no different to the way Apple control the iPhone, iPad and iPod devices.  That is true but there are other devices coming to market that challenge this approach.  The control issue is much wider than simply the management of an App store.  Google store your data on their platform and like all vendors, don’t make things easy when you want to move your data elsewhere.  This is a major issue with the way cloud services are offered today – data inertia.  We have had the issue in Enterprise computing for many years; proprietary database formats, storage arrays that don’t replicate to the competitors products.  In fact the situation is getting worse, as vendors are starting to bring out converged infrastructure that retains the grip over customer data.  Having a laptop that offers local data is still a necessary security blanket.

Then there are the alternatives.  Google are pitching the Chromebook at schools and businesses, making a virtue of the ability to log into any device to access data.  Well we can do that already with Virtual Desktop implementations.  Thin clients are much cheaper than the Chromebook in terms of pure hardware cost.  They also have the same flexible benefit of not storing user data.  The tipping point comes in determining the cost of deploying and managing VDI infrastructure versus using a solution such as Google’s.  As networks get faster and more reliable, why not just use a hosted VDI solution?

The Architect Angle

Google have to do something to meet the challenge from Apple, Amazon (and to a certain extent Microsoft) and others in their core business – delivering data and applications via the web.  I think the “always on” assumption is at this stage flawed and there are plenty of other better solutions already available out there.  Consumers will balk at paying £400 for a device which has no other purpose than to connect to Google.  At that price it is better to purchase a TouchPad or iPad and have access to multiple server providers.  Google have a long uphill struggle ahead of them to get Chromebook into the dominant position Apple enjoy today.

After discussions in the Bloggers’ Lounge and a backstage tour of the main keynote stage, 4pm saw the keynote speech of the day from Leo Apotheker, HP CEO.  This is the guy that looks like Patrick Stewart, talks like Arnold Schwarzenegger and came across as solid and knowledgeable about his products and strategy.  HP’s TouchPad was prominent throughout the session with the emphasis on how it is more suitable for the Enterprise than (presumably) the iPad.  Of course with consumer devices the details matter and the level of usability can’t be gleaned from a canned presentation.  I hope to find the TouchPad stand later today for a review.

The keynote concluded with Don Tapscott, whose insights into technology were thought provoking but possibly came too late in the day to be absorbed fully.  In summary the keynote was good but perhaps lasted too long and the quality of the presenters dipped in places.

By 7pm with keynotes over the Discover Zone was buzzing with 10,000 attendees taking advantage of the “free” food and drink.

Summary

Headline presentations are designed to rouse the troops and get everyone fired up; HP certainly achieved that.  As day 2 begins, it’s time to take stock and look at the substance behind these new announcements.

You can find pictures of HP Discover 2011 on Flickr or via The Storage Architect Facebook page.

• http://www.facebook.com/TheStorageArchitect
• http://www.flickr.com/photos/33818355@N07/sets/72157626902117768

Disclaimer:  As a blogger/attendee, HP paid for my flights, hotel and most meals.  I have not been compensated for my time and am not obliged to blog on the event.  All the opinions are my personal views.

Of course there’s always a catch and in this instance for both of these services it will be in support for the device on which the music plays.  Although the underlying storage for Amazon is generic (an offering called Amazon Cloud Drive), the Amazon Cloud Player (the app to play your music) is only available and supported on specific platforms, notably Windows & Mac PCs, Android, Blackberry and Palm.  There is no Windows phone, iPad, iPod or iPhone support.  In addition, Amazon don’t support WMA or lossless AAC formats.  Google’s offering works on any Flash-enabled device, so no iPod, iPad or iPhone.

It is pretty much guaranteed that Apple will release their own service any day (rumoured to be called iCloud) and this will likely automatically “know” your iTunes purchases and make them available to play.  Apple recently purchased 12PB of storage to cover the iCloud requirements according to speculation in the industry.

Where does this leave the consumer?  The move to cloud music storage is playing out in exactly the same way the music format wars did 10 years ago.  There’s still a confusion of MP3, WMA and AAC formats out there and I’m sure we’ve all got music of varying quality in each one.  I see lack of cross-platform support as nothing more than an effort to retain users to a specific platform.  The effort of moving music collections between providers will be significant and the services are being designed to remove the need to upload content, encouraging users not to store a local copy, so making it difficult to drop a service without spending significant time moving files around and converting them between formats.

Maybe it’s time again to look at independent alternatives.  Previously I tried Zumodrive for the iPhone, which had a music player built in.  This failed to work properly because only the built-in applications were true multi-tasking at the time.  Receiving a text message or other alert dropped the music track, which was incredibly annoying.  Other than Spotify, does anyone have any third party independent suggestions, or do I need to get coding?

As we continue to expose more computing to the Internet (especially with the move to more cloud-based applications), then surely we can expect more incidents of this nature.  Do we really take security seriously enough or is it simply that we don’t understand the nature of how deep we need to ensure security is implemented?  I was in discussion with a company last week that indicated many organisations feel it is necessary only to protect the perimeter against attacks.  This seems remarkably weak as an approach; as soon as the perimeter is breached, then everything is open  and available.  Perhaps we need to elevate the importance of security within IT, because unless we get things right, data exposures and service outages will continue to be a problem.

One other thought comes to mind while thinking about Sony’s dilemma.   The reports (including this from BBC) indicate that Sony are looking to strengthen their network security.  This has necessitated them taking down the whole infrastructure to achieve this.  However one of the original design principles of the Internet that any one section could be damaged and traffic would automatically reroute.  Why are we not designing Internet based applications in this way?  After all, with 70 million users, Sony must be spreading the workload across many physical servers and network infrastructure.  Perhaps we need to be designing with the Internet methodology in mind – breach one component and you don’t get global access; that component can then be taken down until the breach is resolved.

The recent Amazon AWS outage has shown us that traditional application architectures won’t work well in the cloud.  I expect we’ll see a much greater focus on security and application design as organisations struggle to manage embracing Internet-based computing whilst keeping the hackers at bay.