I’ve upgraded The Storage Architect lab servers to R2 SP1 and have been looking at how Dynamic Memory operates.  There are links at the end of this post that provide the official line, but I’ll be discussing my findings and thoughts here first.

Why Is Dynamic Memory needed?

Virtualisation is all about sharing resources – CPU, memory, disk space and so on, in a way that enables those resources to be used more efficiently.  When Windows servers were using 25% or less of their CPU capacity, there was an opportunity to make use of that wasted resource by stacking multiple Windows onto the same hardware using a hypervisor like VMware or Hyper-V.  The virtualisation approach is more suitable to Windows as the O/S doesn’t play well with multiple application installations in one Windows instance.  In addition, for maintenance and other management reasons, it may not be appropriate for multiple business applications to share the same server.

Sharing processor time is relatively easy and is already built into all operating systems that run multiple tasks – so pretty much everything since DOS days has been able to “timeslice”.  Memory however is a little more problematic.  All operating systems (bar a few like z/OS) assume that the memory they detect at boot time is there forever (or at least until the O/S shuts down).  This is a pretty reasonable assumption as hot plugging memory is something I don’t see most server hardware having; the cost of implementing electrical isolation on the motherboard slots doesn’t really justify the cost.

However memory is inevitably overallocated to hosts on their initial creation, even if it is subsequently fully utilised at some stage in the future.  Memory is also the main resource that virtualisation platforms never have enough of, so it makes sense to be able to be more efficient with its use.  Dynamic Memory for Hyper-V enables a host to manage the memory provided to the guests in a more efficient manner by dynamically increasing and decreasing the memory available to each guest.  However there are some caveats:

• The new feature requires the installation of SP1 on Windows Server 2008 R2 running Hyper-V (or Windows Hyper-V Server R2)
• Dynamic memory is only supported on Windows 7, Windows Server 2008 with SP2, Windows Server 2008 R2, Windows Vista with SP1, Windows Server 2003 R2 with SP2 and Windows Server 2003 with SP2 along with either SP1 for that platform or the SP1 version of the integration services.  See here for the specific details.
• Non-supported platforms will continue to use their full allocation of memory.

Integration Services is required on the guest O/S to fix the problem of notifying the O/S that the physical memory allocation has changed.  It looks like for Windows 7 and Windows Server 2008 SP2 that Service Pack 1 onwards effectively provides this capability and should be included in all future releases (although Microsoft would have to confirm that).

As you can imagine, adding and removing what appears to be physical memory isn’t a trivial task.  Remember that as far as the O/S is concerned, the memory it has access to is real physical memory.  Adding memory is no problem; the amount of apparent physical memory is simply increased.  However when taking memory back, the hypervisor needs to be aware that the memory slot may be in use.  In order to take it back from the guest O/S, the memory page would need to be released by paging the contents to disk or another memory page.  I don’t know the specifics of how Hyper-V performs dynamic memory management, but common sense would say that the pages to be retrieved should be the ones not in use by the guest.  Depending on how Windows views physical memory, it may not be possible to remove a non-sequential memory slot; there may be a requirement to remove memory from the end of the address space; it all depends on how Windows memory management algorithms work.  This is something I’ll speculate on after presenting the performance testing results.

Configuration

Dynamic Memory is specified using new options in guest Settings.  See the first screenshot.  Either Static or Dynamic is selected; for Static, a fixed amount of memory is selected; for Dynamic a Startup and Maximum figure is specified. A third parameter, Memory Buffer also needs to be set.  This represents an amount of memory Hyper-V attempts to reserve in addition to the in-use memory; so, if 1000MB have been assigned to a guest and the buffer is 20%, an additional 200MB is reserved.  There’s also an option to weight each guest in terms of priority; so less important virtual machines could be given lower priority when memory is low.

Testing Testing

The second screenshot shows my test environment.  This is a Hyper-V server with 8GB of RAM.  Initially memory is assigned to a number of Windows servers and two non-Windows servers; Memory Demand and Memory Status is shown for those guests running Integration Services.  Running the following command against one of the guests shows that the physical memory assumed by the guest is the initial value, in this case 512MB (screenshot 3).

get-wmiobject Win32_PhysicalMemory -computername hv41-win7-x64

This sort of makes sense, in that the physical memory slots seen at boot time is the value representing startup memory and so this value is also the minimum that the guest can have.  However as I found when testing, the reported memory assigned to a guest increases but never decreases.

To track memory usage, I wrote a small Powershell script that extracts the host and guest memory usage, displaying the results every 10 seconds.  The output is shown in the graph screenshot.  From a starting position of 512MB on the guest, I increased memory demand by starting Internet Explorer multiple times.  This increased memory figures for Guest Used and Physical.  It also increased the memory in use by the Host, all in step with each other.  Around 4 minutes into the test I started other guests on the host.  This immediately increased the Host Used memory sharply and as the Host became under pressure with memory, it caused the Guest’s assigned memory to drop to the bare minimum i.e. what was actually in use.  Once I relieved the pressure (around 9 minutes in) then started to close IE on the Guest, the memory figures returned to levels seen before the test.  There are some interesting pieces of information here:

• The Guest Physical MB figure increased from the base 512MB to towards the maximum assigned (in this case 1024MB).  Even when memory demand in the guest drops, this figure never reduces and is always a high watermark.  There could be a temptation to set the Maximum memory for each guest very high.  I can’t imagine this being wise as increased demand would start paging at the Hyper-V level, impacting all guests.
• At one stage I appeared to have more memory in use than Assigned (around 8 minutes in).  I’m not sure how this could happen, other than for Hyper-V to be paging some of the active guest memory to disk.

Memory usage within Windows was never clear in the first place; Dynamic  Memory just makes things more complex; correctly determining memory usage will require information from both the hypervisor and the guest.  I’m hoping to develop a tool that will make the process of understanding the configuration an easier one.  In the meantime I’d be interested in feedback on how other people have tested this new feature.

• 56% of data on virtual systems is regularly backed up. This seems like a small number but perhaps the term “regularly” is one to consider here.  If environments are cloned and used for test, then perhaps there’s no need to back these environments up as they’re deleted and re-seeded as required.  It would be interesting to know how this figure breaks down by production and non-production environments.
• Only 20% of virtual environments protected by replication or failover technologies. This is a remarkable figure and implies a number of things; array-based replication is still hard to get right with virtualisation; users don’t consider virtual environments “production enough” to replicate them; but probably most important is that this implies there is still a lot of work to be done getting replication right.  Of course VMware with their VAAI initiative are looking to fix this problem and features such as svMotion help, but we know that there is a disconnect between LUN-based VMFS file stores and the granularity required to fail over individual virtual guests to remote locations.
• 60% of virtual environments are not covered in DR plans. The issue here could be similar to the first point above; most of these environments might not be production and so be re-seeded as required, however as virtualisation becomes the norm rather than the exception, DR will become an increasingly important consideration.
• 72% of organisations experience downtime from system upgrades & 70% experience downtime from power outages and failures. There’s always going to be a certain amount of “fat-finger” syndrome in system upgrades but I think these two statistics indicate that there’s a lack of failure planning going into designing infrastructures.  Yes, hardware and software will fail; it always does.  The skill is designing to this and building an infrastructure that meets requirements, including resiliency.

I’ve attached the study in it’s entirety to this post.  I’d be interested in anyone’s feedback on their experiences with the points raised; whether they feel they are valid or not.

Symantec 2010 Disaster Recovery Study

P.S.  Look out for one of Barry’s European colleagues totally missing the point that we are recording a video and trying to explain the GUI to another customer at the same time as we are talking.  Nice.

Click here to view the embedded video.

Static Configurations

Think back to the time before server virtualisation (yes, there was one).  Physically static servers failed over to other physically static servers located in remote data centres.  Once deployed, servers very rarely moved unless there were major physical data centre issues or an upgrade was being performed.  In fact, even when server upgrades occurred, it was typical to acquire a new server and rebuild the application and data on that new hardware to remove any issues with new server drivers, hardware firmware and so on.

Mobility Rules

Server Virtualisation changed all those restrictions.  By abstracting the hardware to generic devices it was possible to place a VMware host on shared storage and have any connected VMware server run that guest.  Very quickly the toolset of features improved to make that host movement transparent and as simple as clicking a button.  This meant servers could be accommodated on any hardware and scaled within the hypervisor.

Storage Restrictions

Unfortunately storage wasn’t so quick to keep up.  The static model of a single physical server in two locations worked well with storage replication technologies that required only one copy of the data to be active at any one time.  Moving an application to another data centre was typically a disaster recovery process and consequently a small outage was acceptable as the storage arrays “failed over” their LUNs to the remote location.  Once the DR issue was solved, the data could be “failed back’ to the original location.  It wasn’t usual to move servers between data centres as a standard operational process.

Virtual Machine environments weren’t well catered for.  Failover of replicated storage wasn’t a transparent process; there was a tradeoff between LUN size and the maximum number of LUNs a hypervisor could support; this made it more complex to architect a Virtual Machine environment with enterprise storage.  Even with vMotion on VMware, where a VM host could be moved transparently across physical hypervisor servers, the storage couldn’t be moved easily.  In fact, the storage restrictions were solved by implementing storage vMotion, rather than have the array achieve the data migration itself.

Step in Federation

This is where storage federation comes in.  It enables any and all copies of a LUN to be updated from multiple locations at the same time.  This means that both the hypervisor and the storage can be load balanced across multiple locations and physical hardware without having to bulk copy the data all the time.  Here’s a simple example to demonstrate the process.

Imagine a 1TB LUN on a storage array acting as a single datastore and supporting 20 production virtual machines.  In the “old” model, moving that LUN to another location would require impacting all 20 virtual machines and making another location the primary target of I/O operations.  There was no ability to go “sub-LUN” and to move individual machines to another location.  Storage vMotion could be used, but that would require replicating the entire VM to another LUN/datastore.  Any attempt to load balance the VM guests would be constrained by the time required to continually move data around the infrastructure.  In addition, moving a replicated VM from one LUN/datastore to another would mean compromsing DR until that LUN had been fully replicated to another location.

Now under a federated environment, that 1TB LUN would be updatable from any location, meaning individual VM hosts on the datastore could be updated from multiple locations at the same time.  This means there is a massive increase in the flexibility of managing workloads across physical locations, offering the ability to workload balance for business and operational benefits.

Summary

Ultimately, federated storage environments will be the future. Products like VPLEX are only the start.  The ultimate goal will mean workloads can be run anywhere, any time.  That will be cool.

My Cloud View

Cloud is not virtualisation.  Virtualisation is virtualisation.  It’s a way of consolidating the physical entities we know today and making them available in the same manner in a virtualised form.  Moving 1000 Windows servers to 1000 virtual machines changes nothing.  You’re still offering IT to customers in the same way – using the concept of that physical entity called a server.  This is a step forward in reducing cost, but not in delivering cloud.

Cloud today is a hybrid of two things; virtualisation & services and service abstraction.  In the first form, Amazon, Rackspace and the others are offering the same physical objects (i.e. servers) but packaged nicely so they can charge you efficiently for them.  What makes this kind of environment a step above virtualisation is that you have no view, concern or interest in how, where or by whom those servers are being provided as long as they meet your requirements for security and availability.  The hosting company is effectively offering you elasticity over and above the basic virtualisation offering you could achieve yourself.  If you need more compute, then they guarantee to provide it and they manage the pain of ensuring resources are available.  Second, is service abstraction; some things are non-server dependent.  For instance; generic databases or email outsourcing.  The hosting company offers you a service, rather than a virtual device.  This leads on to the next step; what Cloud will be in the future.

To see what Cloud Computing can offer, we need to move forward away from the fixed view of infrastructure and start thinking about data and applications.  In the future I want to move away from thinking about fixed data types like files and block devices and think about objects and metadata.  These objects need to be modifiable in a way that gives me consistency, so if I lose a node, or device that’s processing the data, I can recover it from elsewhere and continue working.  I want to process my data in parallel and without geographic constraints, so I can run or move my workload anywhere; presumably to the provider who is offering me best price on that day.

The treatment of data in this form means I need to rethink how applications are written.  I can’t rely on a simple client-server hierarchical architecture but need to move to one where my processing and data is distributed.

Re-writing distributed applications will be the hardest part of delivering Cloud Computing.

Application development in a completely distributed environment creates significant challenges. Here are a few of the most obvious issues;

• How do I ensure consistency (multiple updates in multiple locations)?
• How can I manage performance?
• How can I move my data around easily and quickly?
• How can I back my data up?
• How can I move my application and/or data to make best use of cost savings?
• How can I ensure my data remains secure?

The ultimate Cloud Computing model needs to remove almost all the aspects of infrastructure structure we know today (server types, storage arrays, block devices, NAS) from the customer.  It will offer abstracted programming concepts – store, retrieve, search, update – to process data.  Customers will be charged for their consumption of these resources.  Abstraction will allow applications to be run on (almost) any environment.  These services will still be delivered on infrastructure, but we won’t care or need to care what it looks like.  Now there’s a challenge…

OK, free isn’t quite free – there are a few caveats.  Customers have to pay for maintenance, but other than that, there’s no charge for using UVM on current USP-V deployments.  In addition, customers can “super-size” the offer and also get Hitachi Tiered Storage Manager (HTSM), Hitachi In-System Replication and limited access to Hitachi Dynamic Provisioning (HDP) all for free.

It is unlikely HDS are doing this out of some altruistic concern for their customers.  Clearly they see benefits in driving more business by offering these licences at no cost.  So how can customers benefit?

Traditional Approach

Traditionally, UVM is seen as a tool to manage the migration process or get data into a USP and there’s no question that UVM can be used for these purposes.  Typically, here are some of the costs associated with migration:

• Staff Resource Costs – migrations take time and effort to plan.  The longer they take, the more cost is incurred with change control, planning, co-ordination with other teams and on actually performing the migration, which is typically performed out of hours.
• Software Costs – migrations may require specific software products or tools (e.g UVM or TDMF).

Here are some of the constraints on migration work:

• Limited downtime on servers.
• Data to be migrated out of hours or within certain time frames.
• No host tools (like volume managers) to perform migrations.
• Data migrations are required to disparate locations.
• Data integrity and synchronicity must be maintained (e.g. replicated data must be consistent if an outage occurred during the migration process.

The tradeoff with UVM has always been the cost of the UVM licence compared to the savings which could be made on the above issues by using the product.  For example, UVM could be used to virtualise an entire array (or arrays) behind a USP-V in a single weekend.  Consecutive weekends can then be used to migrate into the USP.  This controlled approach (a) enables the storage managers to perform the migration, freeing up the host teams (b) allows the storage managers to load balance migrations into the USP-V (c) monitor performance as migrations occur (d) enable target servers to be immediately upgraded to new code/driver levels which may not have been supported previously on the older arrays.

Lateral Thinking

Getting UVM for free enables some more lateral thinking:

• Example 1: Migrate out of the USP.  There are lots of examples of moving data into a USP, but what about moving it out?  If you’ve purchased a USP and it’s fully populated with disk and full, how do you save money?  Well, one option is to get a UVM licence and migrate non-critical (and perhaps non-production) data to an externalised array at a lower cost.  Previously this could only be commercially viable if the reduced cost of the external array covered the cost of the UVM licence.  Now there’s no need to worry.
• Example 2: Migrate in and De-Dupe.  With UVM and HDP, external storage arrays can be migrated into the USP and placed on thin provisioned volumes. HDP Zero Page Reclaim removes the “empty” blocks of data during this process.  For certain data profiles, this could provide signficant savings.
• Example 3: A Free Datacentre Migration Tool.  Imagine you’ve got to move data on arrays not currently replicated – but that data needs to reside in another location.  If a pair of USPs are available, why not virtualise the source and target arrays and use the USP to move the data between the two sites?  This could save purchasing replication licences for an array or resolve a problem where replication isn’t possible from certain hardware.
• Example 4: Implement a 3DC Solution.  How about using the USP-V as a three datacentre solution?  Here’s how it works; two USP-Vs are used to replicate data synchronously between two local sites.  The target devices are actually virtualised across a fabric to a third site using UVM.  In this way, data can be moved to a third site without requiring a third copy of data.

With free software, the options are only limited by imagination.  If HDS are prepared to offer software for free then there’s no excuse not to use it.