Virtualization in Grid'5000
Purpose
This page presents how to use KVM on the production environment (with a "non-deploy" reservation). The aim is to permit the execution of virtual machines on the nodes, along with a subnet reservation, which will give you a range of routed IP for your experiment.
In the first part, you will learn the basics of g5k-subnets, which is a prerequisite for the rest of this tutorial. The Quick start explains how to run a VM on the production environment in the minimal number of steps. The next part is optional, it explains in details the contextualization mechanism, which allows you to customize your virtual machines. Finally, in the Multi-site experiment section, we will deploy 2 VMs on 2 sites, and we will measure the network bandwidth between them with iperf.
Prerequisite: Network subnets reservation with g5k-subnets
Users deploying VMs on Grid'5000 need to attribute IP address to them. Each site of Grid'5000 is allocated a /14 block for this purpose, divided in 4 smaller blocks.
OAR can be used to reserve a range of IPs. OAR permits to share the IP resources among users, and avoid the potential IP conflicts at the same time.
Reservation
Subnet reservation through OAR is similar to normal resource reservation.
To reserve 4 /22 subnets and 2 nodes, just type:
You can of course have more complex request. To obtain 4 /22 on different /19 subnets, you can type:
Usage
The simplest way to get the list of your allocated subnets is to use the g5k-subnets
script provided on the head node of the submission.
# g5k-subnets 10.8.0.0 10.8.8.0
Several other printing options are available (-p
option to display the CIDR format, -b
to display broadcast address, -n
to see the netmask, and -a
is equivalent to -bnp
):
# g5k-subnets -a 10.8.0.0/21 10.11.255.255 255.255.252.0 10.11.255.254 10.8.8.0/21 10.11.255.255 255.255.252.0 10.11.255.254
You can also summarize the subnets into a larger one if they are contiguous:
# g5k-subnets -sp 10.8.0.0/20
You can display all the available IP in your reservation, and their associated unique mac addresses, with the following command.
# g5k-subnets -im 10.158.16.1 00:16:3E:9E:10:01 ...
Note | |
---|---|
For detailed information, see the Subnet reservation page. The Grid5000:Network page also describes our organization of the virtual IP space inside Grid'5000. |
Quick start
In this part, we will create a virtual machine in a few steps, and ssh to it.
Job submission
In order to test easily the kvm environment, we use an interactive job, and we reserve one subnet and one node with hardware virtualization capabilities.
Disk image, virtual machine
A disk image containing debian wheezy is available at the following path:
/grid5000/images/KVM/wheezy-x64-base.qcow2
It can be used as a base for more advanced work. For the next steps of this tutorial, copy the disk image to /tmp on the node:
Network configuration
Create a Virtual Interface
For Virtual Machines to use the network, a Tun/Tap interface must be created manually for each of them.
This virtual interface will be attached to your virtual machine, and bridged on the physical machine to the production network. Therefore, the virtual machine will be able to get an IP from the DHCP server and access the network.
A script is available to create automatically this interface on the node:
create_tap
:
- Tun/Tap interfaces are listed by issuing the command
/sbin/ifconfig
.
tap0 Link encap:Ethernet HWaddr 00:16:3e:db:c6:41 inet6 addr: fe80::58ff:a4ff:fe97:c6a8/64 Scope:Link UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:29435 overruns:0 carrier:0 collisions:0 txqueuelen:500 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Note | |
---|---|
- Create one Tun/Tap interface per guest OS. - Use |
Select a MAC address
As seen before, g5k-subnets maintains a correspondence between MAC addresses and IP addresses. The Debian system provided on the disk image is configured to use DHCP and the DHCP server will assign the IP corresponding to the MAC address of the virtual machine.
Consequently, you have to choose an IP in the range you have reserved, and set the MAC address of the VM to the associated MAC address.
You can get the list of available IP, and an associated unique MAC address with the following command.
10.172.0.1 00:16:3E:AC:00:01 10.172.0.2 00:16:3E:AC:00:02 10.172.0.3 00:16:3E:AC:00:03 10.172.0.4 00:16:3E:AC:00:04 10.172.0.5 00:16:3E:AC:00:05 10.172.0.6 00:16:3E:AC:00:06 10.172.0.7 00:16:3E:AC:00:07 10.172.0.8 00:16:3E:AC:00:08 10.172.0.9 00:16:3E:AC:00:09 10.172.0.10 00:16:3E:AC:00:0A ...
Run the guest OS using libvirt
Warning | |
---|---|
A bug breaks libvirt usage on Grid'5000 production environment. See https://intranet.grid5000.fr/bugzilla/show_bug.cgi?id=5257 for details |
Libvirt is a toolkit for managing virtualization servers. Libvirt is also an abstraction layer for different virtualization solutions, including KVM but also Xen and VMWare ESX.
In our case, we use libvirt on top of KVM.
- Create a domain file in XML, describing a virtual machine.
eg : domain.xml
<domain type='kvm'> <name>wheezy</name> <memory>524288</memory> <vcpu>1</vcpu> <os> <type arch="x86_64">hvm</type> </os> <clock offset="localtime"/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/tmp/wheezy-x64-base.qcow2'/> <target dev='vda' bus='virtio'/> <shareable/> </disk> <interface type='ethernet'> <mac address='AA:BB:CC:DD:EE:FF'/> <script path='no'/> <model type='virtio'/> <target dev='tap0'/> </interface> <serial type='pty'> <source path='/dev/ttyS0'/> <target port='0'/> </serial> <console type='pty'> <source path='/dev/ttyS0'/> <target port='0'/> </console> </devices> </domain>
- Now, the guest OS can be started.
- You can also use
virsh
to manage your guest OS:- list the running virtual machines:
virsh list
- open a console on the "wheezy" virtual machine:
virsh console wheezy
- list the running virtual machines:
- At this point, you can repeat the full process and launch several VMs in parallel.
Run the guest OS using the kvm command
You can also use kvm to start the virtual machine:
node :
|
screen kvm -m 512 -hda /tmp/wheezy-x64-base.qcow2 -net nic,model=virtio,macaddr=AA:BB:CC:DD:EE:FF -net tap,ifname=tap0,script=no -nographic |
This is an example command, feel free to adapt it to your use case (The kvm process is launched in a screen
session, if you are not familiar with screen, read its documentation)
SSH to your virtual machine
Finally, you can ssh directly to your VM from anywhere in Grid'5000:
Use contextualization to configure your VMs
This part describes the basic usage of a contextualization, an easy way to dynamically configure your VM.
Mechanism
Contextualization can be used to do any kind of configuration upon VMs boot, in order to configure the virtual machines on boot.. It works like the following :
- Test for the presence of a CD in the CD drive of the VM
- if it exists, mount the CD, test the presence of a script
post-install
, and run it as root
Create a contextualization script
In this example, we will create a simple contextualization script to add your SSH key to enable password-less connection.
- Create a directory to store your contextualization script:
- Copy your SSH public key into it:
- Create a file named "post-install" and add execution rights to it:
- Edit the "post-install" script and write the command needed to add the SSH key to root's authorized_keys file :
#!/bin/sh mkdir -p /root/.ssh cat /mnt/id_rsa.pub >> /root/.ssh/authorized_keys
Note that contextualization files are accessible in the virtual machine inside /mnt directory
- Once you have prepared the content of your contextualization script, you can generate an ISO image :
The file kvm-context.iso
is ready to be attached to a VM, the script included in the iso will be executed and will configure the first network interface at boot time.
Start a VM with contextualization
Use this domain file to start your VM using libvirt:
domain.xml
<domain type='kvm'> <name>wheezy</name> <memory>524288</memory> <vcpu>1</vcpu> <os> <type arch="x86_64">hvm</type> </os> <clock offset="localtime"/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/tmp/wheezy-x64-base.qcow2'/> <target dev='vda' bus='virtio'/> <shareable/> </disk> <disk type='file' device='cdrom'> <source file='/tmp/kvm-context.iso'/> <target dev='vdb' bus='virtio'/> <readonly/> </disk> <interface type='ethernet'> <mac address='AA:BB:CC:DD:EE:FF'/> <script path='no'/> <model type='virtio'/> <target dev='tap0'/> </interface> <serial type='pty'> <source path='/dev/ttyS0'/> <target port='0'/> </serial> <console type='pty'> <source path='/dev/ttyS0'/> <target port='0'/> </console> </devices> </domain>
Start the guest OS and connect it using ssh :
If the contextualization ran correctly, you should not need to enter a password.
Add contextualization to your own VM
The contextualization mechanism is not standard, if you want to use it on your VM, you must adapt it to your image.
- Here is an example of a contextualization script :
#!/bin/bash DEVICE= [ -b /dev/hdb ] && DEVICE=/dev/hdb [ -b /dev/sdb ] && DEVICE=/dev/sdb [ -b /dev/vdb ] && DEVICE=/dev/vdb [ -b /dev/xvdb ] && DEVICE=/dev/xvdb [ -b /dev/sr0 ] && DEVICE=/dev/sr0 if [ -b "$DEVICE" ];then /bin/mount -t iso9660 $DEVICE /mnt 2> /dev/null if [ -f /mnt/post-install ]; then bash /mnt/post-install fi umount /mnt 2> /dev/null fi exit 0
- This contextualization script must be executed during the boot sequence. For instance,
- copy it to /usr/local/bin/init
- add execution rights to /usr/local/bin/init
- to launch the script at VM startup, add:
# KVM contextualization script /usr/local/bin/init
inside /etc/rc.local
, at the end of the file (before the exit 0 if any).
Multi-site experiment
In this part, to illustrate what can be done using Virtual machines on the production environment, we will start two virtual machines on two sites, and make them communicate using the virtualization network.
Reservation
Open 2 terminals, and ssh to the frontends of 2 sites, in this example, it will be the frontend of Luxembourg, and the frontend of Nancy. Then, reserve two virtualization-capable nodes and two subnets on two different sites.
Network configuration
In this part, we will choose an IP for the 2 virtual machines.
Choose a couple of IP & MAC for each VM, in the output of g5k-subnets -im
.
Note that g5k-subnets
returns completely different information on each site. In the following, we assume that you chose 10.144.8.1 (00:16:3e:90:08:01) in Nancy, and 10.172.0.1 (00:16:3e:ac:00:01) in Luxembourg.
Instantiate your VMs
Create the tap interfaces
Copy a standard virtual machine image
Copy the default virtual machine image from /grid5000/images/KVM/wheezy-x64-base.qcow2
to /tmp
Create the domain.xml
file
The domain.xml
file contains the description of your virtual machine. You must adapt it, in order to use a mac address provided by g5k-subnets -im
. The virtual machine will get the IP associated to its mac address.
<domain type='kvm'> <name>wheezy</name> <memory>362144</memory> <vcpu>1</vcpu> <os> <type arch="x86_64">hvm</type> </os> <clock sync="localtime"/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/kvm</emulator> <disk type='file' device='disk'> <driver type='qcow2'/> <source file='/tmp/wheezy-x64-base.qcow2'/> <target dev='vda' bus='virtio'/> <shareable/> </disk> <interface type='ethernet'> <mac address='00:16:3e:90:08:01'/> <script path='no'/> <model type='virtio'/> <target dev='tap0'/> </interface> <serial type='pty'> <source path='/dev/ttyS0'/> <target port='0'/> </serial> <console type='pty'> <source path='/dev/ttyS0'/> <target port='0'/> </console> </devices> </domain> |
<domain type='kvm'> <name>wheezy</name> <memory>362144</memory> <vcpu>1</vcpu> <os> <type arch="x86_64">hvm</type> </os> <clock sync="localtime"/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/bin/kvm</emulator> <disk type='file' device='disk'> <driver type='qcow2'/> <source file='/tmp/wheezy-x64-base.qcow2'/> <target dev='vda' bus='virtio'/> <shareable/> </disk> <interface type='ethernet'> <mac address='00:16:3e:ac:00:01'/> <script path='no'/> <model type='virtio'/> <target dev='tap0'/> </interface> <serial type='pty'> <source path='/dev/ttyS0'/> <target port='0'/> </serial> <console type='pty'> <source path='/dev/ttyS0'/> <target port='0'/> </console> </devices> </domain> |
Launch the two VMs
Enjoy !
SSH in your VMs
Install and run iperf
Finally, we will install iperf
and measure the bandwidth between the two VMs:
- install
iperf
withapt-get
; - then, run
iperf
in server mode (-s
parameter) on one node, and in client mode (-c
parameter) on the other.
root@vm-1:~# iperf -s ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 85.3 KByte (default) ------------------------------------------------------------ [ 4] local 10.144.8.1 port 5001 connected with 10.172.0.1 port 52389 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 1.09 GBytes 938 Mbits/sec |
root@vm-1:~# iperf -c 10.144.8.1 ------------------------------------------------------------ Client connecting to 10.144.8.1, TCP port 5001 TCP window size: 16.0 KByte (default) ------------------------------------------------------------ [ 3] local 10.172.0.1 port 52389 connected with 10.144.8.1 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 1.09 GBytes 938 Mbits/sec |