Virtualization in Grid'5000
Purpose
This page presents how to use KVM on the production environment, with a "non-deploy" reservation. The aim is to permit the execution of virtual machines on the nodes, along with a subnet reservation, which will give you a range of routed IP for your experiment.
In the first part, you will learn the basics of g5k-subnets, which is a prerequisite for the rest of this tutorial. The Quick start explains how to run a VM on the production environment in the minimal number of steps. The next part is optional, it explains in details the contextualization mechanism, which allows you to customize your virtual machines. Finally, in the Multi-site experiment section, we will deploy 2 VMs on 2 sites, and we will measure the network bandwidth between them with iperf.
Prerequisite: Subnet reservation with g5k-subnets
Users deploying VMs on Grid'5000 need to attribute IP address to them. Each site of Grid'5000 is allocated a /14 block for this purpose, divided in 4 smaller blocks.
To answer this problem, there are two solutions:
- use "free to use" IPs from the last /16 block, as described in the virtual network interlink. By using this technique, you are exposed to the risk of IP conflicts with other users.
- reserve a range of IPs with OAR, contained in the 3 /16 other blocks. OAR permit to share the IP resources among users, and avoid the potential IP conflicts at the same time. In the following, we will use this solution.
Note | |
---|---|
For more network related information, see the virtual network interlink page. |
Reservation
Subnet reservation through OAR is similar to normal resource reservation.
To reserve 4 /22 subnets and 2 nodes, just type:
You can of-course have more complex request. To obtain 4 /22 on different /19 subnets, you can type:
Usage
The simplest way to get the list of your allocated subnets is to use the g5k-subnets
script provided on the head node of the submission.
# g5k-subnets 10.8.0.0 10.8.8.0
Different other printing options are available (-p
option to display the CIDR format, -b
to display broadcast address, -n
to see the netmask, and -a
is equivalent to -bnp
):
# g5k-subnets -a 10.8.0.0/21 10.11.255.255 255.255.252.0 10.11.255.254 10.8.8.0/21 10.11.255.255 255.255.252.0 10.11.255.254
You can also summarize the subnets into a larger one if they are contiguous:
# g5k-subnets -sp 10.8.0.0/20
You can display all the available IP in your reservation, and their associated unique mac addresses, with the following command.
# g5k-subnets -im 10.158.16.1 00:16:3E:9E:10:01 ...
Note | |
---|---|
For more precise information, see the Subnet reservation page. |
Quick start
In this part, we will create a virtual machine in a few steps, and ssh to it.
Job submission
In order to test easily the kvm environment, we use an interactive job, and we reserve one subnet and one node with hardware virtualization capabilities.
Disk image, virtual machine
A disk image containing debian squeeze is available at the following path:
/grid5000/images/KVM/squeeze-x64-base.qcow2
It can be used as a base for more advanced work. For the next steps of this tutorial, copy the disk image to /tmp on the node:
Network configuration
In order to use the network with kvm, a Tun/Tap interface must be created for each virtual machines. This virtual interface will be attached to your virtual machine, and bridged on the production network. Therefore, the virtual machine will be able to get an IP from the DHCP server and access the network.
A script is available to create automatically this interface on the node:
create_tap
:
- Tun/Tap interfaces are listed by issuing the command
/sbin/ifconfig
.
tap0 Link encap:Ethernet HWaddr 00:16:3e:db:c6:41 inet6 addr: fe80::58ff:a4ff:fe97:c6a8/64 Scope:Link UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:29435 overruns:0 carrier:0 collisions:0 txqueuelen:500 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Note | |
---|---|
- Create one Tun/Tap interface per guest OS. - Use |
Generate the contextualization iso file
This file contains a script which will set the network configration of your VM.
First, choose an IP in the range you have reserved. The command g5k-subnets
will give you more information.
10.172.0.0/22 10.175.255.255 255.252.0.0 10.175.255.254 10.172.0.0 dns.luxembourg.grid5000.fr 172.16.191.101
You can get the list of available IP, and an associated unique mac address with the following command.
10.172.0.1 00:16:3E:AC:00:01 10.172.0.2 00:16:3E:AC:00:02 10.172.0.3 00:16:3E:AC:00:03 10.172.0.4 00:16:3E:AC:00:04 10.172.0.5 00:16:3E:AC:00:05 10.172.0.6 00:16:3E:AC:00:06 10.172.0.7 00:16:3E:AC:00:07 10.172.0.8 00:16:3E:AC:00:08 10.172.0.9 00:16:3E:AC:00:09 10.172.0.10 00:16:3E:AC:00:0A ...
We will now suppose that you want to generate a virtual machine with the IP 10.172.0.1. Do not forget to adapt all the following commands and examples to your subnet reservation.
With the following command, we will create a contextualization iso file:
Run the guest OS using the kvm command
Start the virtual machine with the kvm command. The following command is just an example, feel free to adapt it to your use case.
The kvm process is launched in a screen
session, if you are not familiar with screen, read its documentation.
node :
|
screen kvm -m 512 -hda /tmp/squeeze-x64-base.qcow2 -cdrom /tmp/kvm-context-10.172.0.1.iso -net nic,model=virtio,macaddr=00:16:3E:AC:00:01 -net tap,ifname=tap0,script=no -nographic |
Or, use libvirt
Libvirt is a toolkit for managing virtualization servers. Libvirt is also an abstraction layer for different virtualization solutions, including KVM but also Xen and VMWare ESX.
In our case, we use libvirt on top of KVM.
- Create a domain file in XML, describing a virtual machine.
eg : domain.xml
<domain type='kvm'> <name>squeeze</name> <memory>524288</memory> <vcpu>1</vcpu> <os> <type arch="x86_64">hvm</type> </os> <clock sync="localtime"/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/kvm</emulator> <disk type='file' device='disk'> <driver type='qcow2'/> <source file='/tmp/squeeze-x64-base.qcow2'/> <target dev='vda' bus='virtio'/> <shareable/> </disk> <disk type='file' device='cdrom'> <source file='/tmp/kvm-context-10.172.0.1.iso'/> <target dev='vdb' bus='virtio'/> <readonly/> </disk> <interface type='ethernet'> <target dev='tap0'/> <script path='no'/> <model type='virtio'/> <mac address='00:16:3E:AC:00:01'/> </interface> <serial type='pty'> <source path='/dev/ttyS0'/> <target port='0'/> </serial> <console type='pty'> <source path='/dev/ttyS0'/> <target port='0'/> </console> </devices> </domain>
Note | |
---|---|
Adapt this file to your case, you must change the mac address and the path of the contextualization iso file |
- Now, the guest OS can be started.
- You can also use
virsh
to manage your guest OS:- list the running virtual machines:
virsh list
- open a console on the "squeeze" virtual machine:
virsh console squeeze
- list the running virtual machines:
- At this point, you can repeat the full process and launch several VMs in parallel.
SSH to your virtual machine
Finally, you can ssh directly to your VM:
KVM contextualization explained
This part describes the basic usage of a contextualization iso file with KVM, in order to configure the virtual machines (especially the network side). The contextualization script can be easily extended for other purposes.
Mechanism
Principe
Contextualization mechanism works like the following :
- Test for the presence of a CD in the CD drive of the VM
- if it exists, mount the CD, test the presence of a script
post-install
, and run it as root - if it does not exist, use
dhcp
on the first network interface.
Installation
The contextualization mechanism is not standard, if you want to use it on your vm, you must copy and adapt a few scripts.
- The contextualization script is executed during the boot sequence.
This script can be placed in /etc/rc.local
, at the end of the file (before the exit 0 if any).
# KVM contextualization script /usr/local/bin/init
- Here is an example of a contextualization script :
#!/bin/bash DEVICE= [ -b /dev/hdb ] && DEVICE=/dev/hdb [ -b /dev/sdb ] && DEVICE=/dev/sdb [ -b /dev/vdb ] && DEVICE=/dev/vdb [ -b /dev/xvdb ] && DEVICE=/dev/xvdb [ -b /dev/sr0 ] && DEVICE=/dev/sr0 if [ -b "$DEVICE" ];then /bin/mount -t iso9660 $DEVICE /mnt 2> /dev/null if [ -f /mnt/post-install ]; then bash /mnt/post-install fi umount /mnt 2> /dev/null else ifup eth0 fi exit 0
Generate your contextualization iso file
This part explain how to generate the contextualization iso file manually.
Note that the manipulation can be automated with the script g5k-vm
.
- A basic example of contextualization iso file is available at
/grid5000/images/KVM/kvm-context.tgz
on each site.- the entry point is the file
kvm-context/post-install
; - this example configure the network interface using a static IP address and the network information provided in the file
kvm-context/network/common
; - you can customize that iso file for your experiments.
- the entry point is the file
- Use the tool
genisoimage
to generate the contextualization iso file.
Retrieve the file kvm-context.tgz
Uncompress the file /grid5000/images/KVM/kvm-context.tgz
in your home directory. This tarball contains an example of contextualization script.
Adapt the network configuration in the contextualization script
- The contextualization script will apply your network settings.
- The command
g5k-subnets
will give you all the needed network information related to your subnet reservation.
- Choose an IP and a mac address in your range:
10.172.0.1 00:16:3E:AC:00:01 10.172.0.2 00:16:3E:AC:00:02 10.172.0.3 00:16:3E:AC:00:03 10.172.0.4 00:16:3E:AC:00:04 10.172.0.5 00:16:3E:AC:00:05 10.172.0.6 00:16:3E:AC:00:06 10.172.0.7 00:16:3E:AC:00:07 10.172.0.8 00:16:3E:AC:00:08 10.172.0.9 00:16:3E:AC:00:09 10.172.0.10 00:16:3E:AC:00:0A ...
- Following the information displayed, you should adapt the file
./kvm-context/common/network
.
This file is used by the script ./kvm-context/distributions/debian/00_network
in order to configure the network interface for your VM.
Here is an example:
IPADDR=10.172.0.1 MACADDR=00:16:3e:ac:00:01 GATEWAY=10.175.255.254 NETWORK=10.172.0.0 BROADCAST=10.175.255.255 NETMASK=255.252.0.0 NAMESERVER=172.16.191.101 DOMAIN=luxembourg.grid5000.fr SEARCH=luxembourg.grid5000.fr
Generate the iso file
Once you have prepared the content of the iso file for the contextualization, you can generate it in /tmp
on the node.
The file (replace $ISOFILE
by a file name) is ready to be attached to a VM, the script included in the iso will be executed and will configure the first network interface at boot time.
Multi-site experiment
In this part, to illustrate what can be done using Virtual machines on the production environment, we will start two virtual machines on two sites, and make them communicate using the virtualization network.
Reservation
Open 2 terminals, and ssh to the frontends of 2 sites, in this example, it will be the frontend of Luxembourg, and the frontend of Nancy. Then, reserve two virtualization-capable nodes and two subnets on two different sites.
Contextualization
In this part, we will create a contextualization iso, as seen in the previous tutorial KVM.
Choose an IP for each VM, in the output of g5k-subnets -im
.
Note that g5k-subnets
returns completely different information on each site. In the following, we assume that you chose 10.144.8.1 in Nancy, and 10.172.0.1 in Luxembourg.
Then, generate your contextualization iso file:
Instantiate your VMs
Create the tap interfaces
Copy a standard virtual machine image
Copy the default virtual machine image from /grid5000/images/KVM/squeeze-x64-base.qcow2
to /tmp
Create the domain.xml
file
The domain.xml
file contains the description of your virtual machine. You must adapt it, in order to use a mac address provided by g5k-subnets -im
. The virtual machine will get the IP associated to its mac address.
<domain type='kvm'> <name>squeeze</name> <memory>362144</memory> <vcpu>1</vcpu> <os> <type arch="x86_64">hvm</type> </os> <clock sync="localtime"/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/kvm</emulator> <disk type='file' device='disk'> <driver type='qcow2'/> <source file='/tmp/squeeze-x64-base.qcow2'/> <target dev='vda' bus='virtio'/> <shareable/> </disk> <disk type='file' device='cdrom'> <source file='/tmp/kvm-context-10.144.8.1.iso'/> <target dev='vdb' bus='virtio'/> <readonly/> </disk> <interface type='ethernet'> <target dev='tap0'/> <script path='no'/> <model type='virtio'/> <mac address='00:16:3e:90:08:01'/> </interface> <serial type='pty'> <source path='/dev/ttyS0'/> <target port='0'/> </serial> <console type='pty'> <source path='/dev/ttyS0'/> <target port='0'/> </console> </devices> </domain> |
<domain type='kvm'> <name>squeeze</name> <memory>362144</memory> <vcpu>1</vcpu> <os> <type arch="x86_64">hvm</type> </os> <clock sync="localtime"/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/kvm</emulator> <disk type='file' device='disk'> <driver type='qcow2'/> <source file='/tmp/squeeze-x64-base.qcow2'/> <target dev='vda' bus='virtio'/> <shareable/> </disk> <disk type='file' device='cdrom'> <source file='/tmp/kvm-context-10.172.0.1.iso'/> <target dev='vdb' bus='virtio'/> <readonly/> </disk> <interface type='ethernet'> <target dev='tap0'/> <script path='no'/> <model type='virtio'/> <mac address='00:16:3e:ac:00:01'/> </interface> <serial type='pty'> <source path='/dev/ttyS0'/> <target port='0'/> </serial> <console type='pty'> <source path='/dev/ttyS0'/> <target port='0'/> </console> </devices> </domain> |
Launch the two VMs
Enjoy !
SSH in your VMs
Install and run iperf
Finally, we will install iperf
and measure the bandwidth between the two VMs:
- install
iperf
withapt-get
; - then, run
iperf
in server mode (-s
parameter) on one node, and in client mode (-c
parameter) on the other.
root@vm-1:~# iperf -s ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 85.3 KByte (default) ------------------------------------------------------------ [ 4] local 10.144.8.1 port 5001 connected with 10.172.0.1 port 52389 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 1.09 GBytes 938 Mbits/sec |
root@vm-1:~# iperf -c 10.144.8.1 ------------------------------------------------------------ Client connecting to 10.144.8.1, TCP port 5001 TCP window size: 16.0 KByte (default) ------------------------------------------------------------ [ 3] local 10.172.0.1 port 52389 connected with 10.144.8.1 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 1.09 GBytes 938 Mbits/sec |