In a previous post on vSphere integration with OpenStack Nova Compute, I gave an overview of the OpenStack Nova Compute project and how VMware vSphere integrates with that particular project. In this post, I want to review Compute resource scheduling, a.k.a. nova-scheduler in OpenStack and Distributed Resource Scheduler (DRS) in vSphere. I want to show how DRS compares with nova-scheduler and the impact of running both as part of an OpenStack implementation. Note that I can’t be completely exhaustive in this blog post, but would recommend everyone read the following:
Nova Compute uses the nova-scheduler service to determine which compute node should be used when a VM instance is to be launched. The nova-scheduler, like vSphere DRS, makes this automatic initial placement decision using a number of metric and policy considerations, such as available compute resources and VM affinity rules. Note that unlike DRS, the nova-scheduler does not do periodic load-balancing or power-management of VMs after the initial placement. The scheduler has a number of configurable options that can be accessed and modified in the nova.conf file, the primary file used to configure nova-compute services in an OpenStack cloud.
scheduler_driver=nova.scheduler.multi.MultiScheduler compute_scheduler_driver=nova.scheduler.filter_scheduler.FilterScheduler scheduler_available_filters=nova.scheduler.filters.all_filters --> scheduler_default_filters=AvailabilityZoneFilter,RamFilter,ComputeFilter --> least_cost_functions=nova.scheduler.least_cost.compute_fill_first_cost_fn compute_fill_first_cost_fn_weight=-1.0
The two variables I want to focus on are “schedule_default_filters” and “Least_cost_functions.” They represent two algorithmic processes used by the Filter Scheduler to determine initial placement of VMs (There are two other schedulers, the Chance Scheduler and the Multi Scheduler, that can be used in place of the Filter Scheduler; however, the Filter Scheduler is the default and should be used in most cases). These two process work together to balance workloads across all existing compute nodes at VM launch, much in the same way the Dynamic Entitlements and Resource Allocation Settings are used by DRS to balance VMs across an ESXi cluster.
When a request is made to launch a new VM, the nova-compute service contacts the nova-scheduler to request placement of the new instance. The nova-scheduler uses the scheduler, by default the Filter Scheduler, named in the nova.conf file to determine that placement. First, a filtering process is used to determine which hosts are eligible for consideration and an eligible host list is created and then a second algorithm, Costs and Weights (described later), is applied against the list to determine which compute node is optimal for fulfilling the request.
The Filtering process uses the
scheduler_available_filters configuration option in
nova.conf to determine what filters will be used to filter out ineligible compute nodes and to create the eligible hosts list. By default, there are three filters that are used:
An example of how the Filtering process may work is that host 2 and host 4 above may have been initially filtered out for any combination of the following reasons, assuming the default filters were applied:
There are a number of other filters that can be used along with or in place of the default filters; some of them include:
The full list of filters are available in the Filters section of the “OpenStack Compute Administration Guide.” The Nova-Scheduler is flexible enough that customer filters can be created and multiple filters can be applied simultaneously.
Next, the Filter Scheduler takes the hosts that remain after the filters have been applied and applies one or more cost function to each host to get numerical scores for each host. Each cost score is multiplied by a weighting constant specified in the
nova.conf config file. Details on the algorithm used are detailed in the “OpenStack nova-scheduler and its algorithm” I referenced previously. The weighting constant configuration option is the name of the cost function, with the
_weight string appended. Here is an example of specifying a cost function and its corresponding weight:
least_cost_functions=nova.scheduler.least_cost.compute_fill_first_cost_fn,nova.scheduler.least_cost.noop_cost_fn compute_fill_first_cost_fn_weight=-1.0 noop_cost_fn_weight=1.0
There are three cost functions available; any of the functions can used alone or in any combination with the other functions.
The Cost and Weight function is analogous to the Share concept in vSphere DRS.
In the example above, if we choose to only use the nova.scheduler.least_cost.compute_fill_first_cost_fn function and set the weight to compute_fill_first_cost_fn_weight=1.0, we would expect the following results:
Now that we’ve reviewed how Nova compute schedules resources and how it works compared with vSphere DRS, let’s look at how DRS and the nova-scheduler work together when integrating vSphere into OpenStack. Note that I will not be discussing Nova with the ESXDriver and standalone ESXi hosts; in that configuration, the filtering works the same as it does with other hypervisors. I will, instead, focus on Nova with the VCDriver and ESXi clusters managed by vCenter.
Since vCenter abstracts the ESXi hosts from the nova-compute service, the nova-scheduler views the cluster as a single host with resources amounting to the aggregate resources of all ESXi hosts in the cluster. This has two effects:
The latter effect can cause issues with how VMs are scheduled/distributed across a multi-compute node OpenStack environment. Let’s take a look at two use cases where the nova-scheduler may be impeded in how it attempts to schedule resources; we’ll focus on RAM resources, using the environment shown above, and assume we are allowing the default of 50 percent overcommitment on physical RAM resources.
So how do we design around this issue? Unfortunately, the VCDriver is new to OpenStack and there does not seem to be a great deal of documentation or recommended practices available. I hope this changes shortly and I plan to help as best as I can. I’ve been speaking with the VMware team working on OpenStack integration and I hope we’ll be able to collaborate together to put out documentation and recommended practices. VMware is also continuing to improve on the code it’s already contributed and I expect some of these issues will be addressed in future OpenStack releases; I also expect that additional functionality will be added to Nova compute as OpenStack continues to mature.
Want to know more about Kenneth Hui and his thoughts on OpenStack? Check out this Q&A interview.
Heading to VMworld 2013, August 25 to August 29 in San Francisco? We are too.
Come check us out at booth No. 2404 on the expo floor – we’re excited to talk to you about new additions to our VMware® product line and the future of the Rackspace Hybrid Cloud. We’ll also have technical experts in the booth to answer all of your questions about everything from Rackspace Replication Manager (powered by VMware SRM to VMware to OpenStack.
While you’re here, play our “Unlock the Cloud” game for a chance to win prizes.
And be sure to see Rackspace Enterprise Cloud Strategist Paul Croteau and VMware’s Bryan Evans present “DR to the Cloud with VMware Site Recovery Manager and Rackspace Disaster Recovery Planning Services” at 4 p.m. PDT on Monday, August 26.