Booting and Using Virtual Machines on Grid'5000

See also: [| Large Scale Management of Virtual Machines(HEMERA)]

Overview

The goal of this tutorial is to introduce the deployment of a large amount of virtual machines over the Grid'5000 platform.

After a short precision about Grid'5000 specifications and the requirements of this session, users will be presented a set of scripts and tools designed over the Grid'5000 software stack to deploy and interact with a significant number of virtual machines. Our largest experiment and a few related tips are also described.

Once deployed, these instances can then be used at the user convenience in order to investigate particular concerns such as the impact of migrating a large amount of VMs simultaneously or the study of new proposals dealing with VM images management.

Grid'5000 specifications

When booting KVM instances over the platform, we need physical hosts that supports hardware virtualisation.

During the deployment, we are in charge of configuring the IP network used by virtual machines throught network isolation capabilities of Grid'5000 and the subnet reservation system. It enables the use of an IP network ranging from /22 to /16 subnets, and ensures the communications between all the instances.

Deployment

Booking the resources

The first step is to retrieve the last version of the scripts (the dynVM__TEMPLATE__ is located on sophia site)

frontend:

scp -r /home/alebre/dynVM__TEMPLATE__/ DESTINATION_FOLDER

Move to the folder containing the code

frontend:

cd DESTINATION_FOLDER

Book the g5k resources according to the desired time and duration of your experiment.

We will now consider as master site the site from which you book the ressources and executes the scripts.

This script will return you an OAR that includes the reservation of the nodes, a virtual network and a subnet

frontend:

./getmaxgridnodes.sh 2012-09-04 19:00:00 03:00:00

Execute the oar request returned by getmaxgridnodes.sh (don't forget to redirect it as shown in the example) The master site is sophia in the following example.

Warning : Take care of using clusters/site that offers hardware virtualization support (https://api.grid5000.fr/sid/ui/quick-start.html)

frontend:

oargridsub -t deploy -w 3:00:00 -s "2012-24-07 19:00:00" sophia:rdef="{\\\\\\\"type='kavlan-global'\\\\\\\"}/vlan=1+/nodes=5+/slash_18=1",rennes:rdef=/nodes=5,nancy:rdef=/nodes=5,lille:rdef=/nodes=5 > ~/oargrid.out

Deploying and configuring the physical machines

Move to the Flauncher directory

frontend:

cd Flauncher

Get the list of nodes and then connect the user to job

frontend:

./grid_get_nodes.sh

You are now connected to the grid OAR job

Display the list of nodes (optional)

frontend:

less ~/gridnodes-uniq

Move to the Flauncher directory

frontend:

cd DESTINATION_FOLDER/Flauncher

Deploy the nodes

frontend:

./grid_deploy_nodes.sh

Deploy the vlan and set the hypervisor (Warning : Use lowercase letter and repeat the master site as the first site)

frontend:

./vlan_auto_launch.sh sophia "sophia nancy lille rennes"

We use a service node during the process to perform the remote operation on the nodes of the infrastructure.

Retrieve the service node

service_node=$(sed -n '/sophia/p' ./log/machines-list.txt | head -n1)

Connect to the service node

frontend:

ssh root@$service_node

Figure 1: FLauncher Process

Usage

Retrieving the infrastructure informations

The service node contains all the informations about the deployed infrastructure.

Those informations can be accessed through various getters functions according to the desired informations.

Example : Retrieving the mapping between the reserved IP and the nodes

service node:

./get_names_and_ip_from_hosts.sh

Please consult the README file for complete description of the available getters.

service node:

less README

Creation of the Virtual Machines

Create a single virtual machine on a host

It is possible to create a single virtual machine by using the following script

The creation needs an IP adress(IP) , an amount of RAM memory(MEMORY), the current site and node and a arbitrary ID.

on physical host:

./create_vm_local.sh -i IP -m MEMORY -n SITE -s NODE -v ID

Create all the virtual machines from a service node

When the infrastructure require a large amount of virtual machines with a specific mapping, it is necessary to describe the desired infrasctructure to enable an automatic and remote deployment.

#list of nodes
suno-3.sophia.grid5000.fr 4 20 8
suno-6.sophia.grid5000.fr 4 20 8
suno-9.sophia.grid5000.fr 4 20 8
#list of VMs
vm00 1 1 1 2147483647
vm01 1 1 1 2147483647
vm02 1 1 1 2147483647
vm03 1 1 1 2147483647
vm04 1 1 1 2147483647
vm05 1 1 1 2147483647
#initial configuration
suno-3.sophia.grid5000.fr vm00 vm03
suno-6.sophia.grid5000.fr vm01 vm04
suno-9.sophia.grid5000.fr vm02 vm05
#end of configuration

In this particular configuration file, we declare 3 physical nodes and 6 virtual machines.

The last part of the configuration describes a simple mapping of 2 virtual machines on each node.

(The nodes must designate the physical hostnames of your infrastructure)

We can now create virtual machines on each host according to the specified configuration file, from the service node.

service node:

./remote_create_vms.sh CONF_FILE

All the virtual instances are started at the same time, using a hierarchical structure among the physical nodes.The correlation between names and IPs is stored in a dedicated file propagated on each physical node. This allows us to identify and communicate with all the virtual machines.

At the end of the operation, the instances previously described are booted and available for experiments.

Interaction with the Machines

We need to be able to control,monitor and communicate with both the host OSes and the guest instances spread across the infrastructure at any time.

For that purpose, the following scripts are based on hierarchical communication structures to ensure a large-scale communication with the physical and/or virtual instances.

Execute remote commands

Execute a command on a list of nodes using a tree distribution

service node:

./mykash.sh nodefile command timeout

Copy files

Copy a file on a list of node using a tree distribution

service node:

./mykaput.sh nodefile file_to_upload remote_path tree_arity

In Practice : 10240 Virtual Machines on 512 Physical Hosts

Considering that physical machines must support hardware virtualization to start KVM instances, the largest experiment that has been conducted up to now involved 10240 KVM instances upon 512 nodes through 4 sites and 10 clusters. The whole setup is performed in less than 30 minutes with about 10 minutes spent on the deployment of the nodes, 5 minutes for the installation and configuration of the required packages on the physical hosts, while the rest is dedicated to the booting of the virtual machines. The result of that work opens doors to the manipulation of virtual machines thoughout a distributed infrastructure like traditionnal operating systems handle process on a local node.

Useful Tips

Booking grid resources

• Provide the request that retrieves the maximum number of nodes that is available during the time slot defined on specific clusters

frontend:

disco -s '2012-07-24 19:00:00' -w 1 griffon graphene

Deployments

• Ensure a minimum amount of deployed nodes

To ensure that 95% (rounded down) of the reserved nodes are correctly deployed (3 attempts max), instead of running:

frontend:

kadeploy3 -e squeeze-x64-prod -f $OAR_NODEFILE -k

Run:

NB_NODES=$(sort -u $OAR_NODE_FILE | wc -l)
MIN_NODES=$(($NB_NODES * 95/100))
/grid5000/code/bin/katapult3 --deploy-env squeeze-x64-prod --copy-ssh-key --min-deployed-nodes $MIN_NODES --max-deploy-runs 3

Communication

• About the Saturation of ARP tables

ARP_table_overflow_tuning

TCP_bandwidth_tuning

Tools

• Network isolation over Grid'5000 : KaVLAN
• Booking a range of IP adresses : Subnet Reservation System
• Deployment of nodes : Kadeploy3
• Execution Remote commands : TakTuk

Contact

• Adrien Lèbre, Associate Professor at Mines Nantes, Ecole des mines de Nantes, Ascola Team | mail
• Daniel Balouek, Hemera Engineer, ENS Lyon, Inria, Avalon Team | mail
• Flavien Quesnel, PhD student, Ecole des mines de Nantes, Ascola Team | mail