Network reconfiguration tutorial
![]() |
Note |
---|---|
This page is actively maintained by the Grid'5000 team. If you encounter problems, please report them (see the Support page). Additionally, as it is a wiki page, you are free to make minor corrections yourself if needed. If you would like to suggest a more fundamental change, please contact the Grid'5000 team. |
Introduction
The aim of this tutorial is present a method to configure a custom network in Grid'5000, using KaVLAN.
KaVLAN is a tool of Grid'5000, which allows a user to manage VLANs in the platform. Behind the scene, KaVLAN actually changes the network switchs configurations, to set the VLAN membership (VLAN ID) for the ports which corresponds to the network interface of one of more nodes. This method provides a network layer 2 isolation for experiments. Three kinds of KaVLAN VLANs are available on Grid'5000. You can find more information in the KaVLAN page.
In this Tutorial, we will only use global and local VLANs (no routed VLAN).
Initially, we will set up a simple topology with 2 VLANs, a global one and a local one. For each of those VLANs, there will be 2 nodes (at least). One of those nodes must have 2 interfaces, one of which being configured in the VLAN.
Set up topology
Reservations
A global VLAN spread all over Grid'5000, so we need to reserve it on one site only. We will use the site of Rennes for that.
![]() |
rennes:frontend :
|
oarsub -l {"type='kavlan-global'"}/vlan=1+{"type='kavlan-local'"}/vlan=1+{"cluster='paravance'"}/nodes=3,walltime=3 -I -t deploy |
With this reservation, we have 1 kavlan-local, 1 kavlan-global and 3 nodes on paravance cluster. Paravance is a cluster of nodes with 2 interfaces.
Get your Vlans ID :
You can know which VLAN ID is global and which is local, using the following table:
KaVLAN name in OAR | type | first id | last id |
---|---|---|---|
kavlan-local | local | 1 | 3 |
kavlan | routed | 4 | 9 |
kavlan-global | global | 10 | 21 |
(see KaVLAN for more details)
Deployment
Now we will deploy our nodes with debian jessie minimal :
During the deployment you can reserve and deploy a node on Nancy in another terminal :
We will install 'at' and 'tcpdump' on each node using TakTuk :
![]() |
rennes:node :
|
taktuk -s -l root -f $OAR_FILE_NODES broadcast exec [ "apt-get update; apt-get --yes install at tcpdump" ]
|
Since jessie, the default sshd configuration doesn't allow password authentication, then you can't connect to a deployed node from an other node. There are two solutions:
kaconsole
Kaconsole is a tool provided in Grid'5000, basically it allows to connect to a node the same way you would connect to it with a screen and a keyboard.
So, you can connect on the node with the credentials "root":"grid5000".
Add SSH key pair on each node
You can just use this small script from the reservation prompt :
#! /bin/bash temp=$(mktemp -d XXXXX) ssh-keygen -t rsa -f $temp/id_rsa -P "" for i in $(uniq $OAR_NODEFILE) ; do scp $temp/id_rsa root@$i:.ssh/id_rsa && ssh-copy-id -i $temp/id_rsa.pub root@$i done rm -r $temp
It will generate a temporary RSA key pair, and copy the private one as well as allow the public one on each node.
Network configuration
Nancy
We will put a static IP on our reserved node in Nancy and put it in the global VLAN we reserved. To do so, we configure the network interface (we assume here that is is eth0) with the following in /etc/network/interfaces:
auto eth0 iface eth0 inet static address 192.168.1.1 netmask 255.255.255.0
Then exit from the node, and reconfigure the network and vlan of the node from the frontend:
![]() |
nancy:frontend :
|
ssh root@Hostname_node_nancy "apt-get --yes install at && echo 'service networking restart' | at now + 1 minute" && kavlan -s -i Global_Vlan_Id -m Hostname_Node_Nancy |
Rennes
Let's call our 3 nodes on Rennes node1, node2 and node3. Of course during the TP node1=<hostname of your first node>. For example node1=paravance-23.
We will put node1 in the local VLAN and restart the networking service to get a new IP address (there is a DHCP server in kavlan-local).
![]() |
rennes:frontend :
|
ssh root@node1 "echo 'service networking restart' | at now + 1 minute" && kavlan -s -i Local_Vlan_Id -m node1 |
![]() |
Warning |
---|---|
Later we will configure an OpenVSwitch on node2, so install it with 'apt-get install openvswitch-switch' before putting node2 in another VLAN than (production) DEFAULT one |
Now we can put node2 between the local and global VLANs. We have to set up the second interface. According to the API, eth1 is linked : https://api.grid5000.fr/sid/sites/rennes/clusters/paravance/nodes/paravance-1.json?pretty
(bpichot) I don't understand the last sentence…
So we have to add it to /etc/network/interfaces on node2 :
auto eth1 iface eth1 inet static address 192.168.1.2 netmask 255.255.255.0
We need to put the first interface in the local-vlan and the second in the global :
![]() |
rennes:frontend :
|
ssh root@node2 "echo 'service networking restart' | at now + 1 minute" && kavlan -s -i Local_Vlan_Id -m node1 && kavlan -s -i Global_Vlan_Id -m node2-eth1 |
Now we have :
[prod <-]--kavlan-ID--[-> local-vlan] [local-vlan <--node1-->] [local-vlan <-]--node2--[-> global-vlan] [global-vlan <--node_nancy-->]
(bpichot) looks like there is a node named "kvlan-ID" spanning accross 2 VLANs (+ still no definition of node_nancy)
Now you should be able to connect in SSH on node2 from kavlan gateway, and ping 192.168.1.1 (node_nancy).
Routing
node1 and node_nancy are in two different VLANs, so currently it's impossible to send a packet between node1 and node_nancy. To do this, we will set a route between the local VLAN and the global VLAN (192.168.1.0/24).
![]() |
Note |
---|---|
To know the network address in the local VLAN, connect to node1, node2 or kavlan-ID and enter the command : ip route |
Packets can't go from a network to another without a router between the VLANs, this will the role of our gateway (node2). First, we have to enable routing on node2:
Now, if an IP packet with a destination in a known network is sent to node2, it will be forwarded.
But node1 deosn't know how to reach node_nancy's network, and vice versa. Let's add a route to each node:
Topology test
We will check if packets are able to pass through our gateway.
Open two terminals :
On the first terminal you should see that the ICMP packets are forwarded on node2 :
IP 192.168.200.7 > 192.168.1.1: ICMP echo request, id 4270, seq 1, length 64 IP 192.168.1.1 > 192.168.200.7: ICMP echo reply, id 4270, seq 1, length 64 IP 192.168.200.7 > 192.168.1.1: ICMP echo request, id 4270, seq 2, length 64 IP 192.168.1.1 > 192.168.200.7: ICMP echo reply, id 4270, seq 2, length 64 IP 192.168.200.7 > 192.168.1.1: ICMP echo request, id 4270, seq 3, length 64 IP 192.168.1.1 > 192.168.200.7: ICMP echo reply, id 4270, seq 3, length 64
If ip forwarding is disabled on node2, nothing will happen in tcpdump and ping will fail !
We can check the route with traceroute from node1:
Which should show two hops, the intermediary router and the target:
traceroute to 192.168.1.1 (192.168.1.1), 30 hops max, 60 byte packets 1 192.168.200.8 (192.168.200.8) 0.136 ms 0.122 ms 0.115 ms 2 192.168.1.1 (192.168.1.1) 25.612 ms 25.617 ms 25.611 ms
The first jump is from node1 to node2, and the second node is from node2 to node_nancy.
With tcpdump we can check the complete isolation of nodes from the production VLAN (and from any other one) :
On node1 (in 12 seconds):
14:26:15.107927 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 82bd.8c:60:4f:47:6c:bc.808e, length 43 14:26:17.109436 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 82bd.8c:60:4f:47:6c:bc.808e, length 43 14:26:19.108669 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 82bd.8c:60:4f:47:6c:bc.808e, length 43 14:26:21.108675 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 82bd.8c:60:4f:47:6c:bc.808e, length 43 14:26:23.108669 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 82bd.8c:60:4f:47:6c:bc.808e, length 43 14:26:25.108654 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 82bd.8c:60:4f:47:6c:bc.808e, length 43 14:26:27.108674 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 82bd.8c:60:4f:47:6c:bc.808e, length 43
The only frames received are spanning tree frames, from the switch.
In comparison, this is what can be captured on a node in the production VLAN, in only one second:
14:27:43.920934 IP paravance-60.rennes.grid5000.fr.38784 > dns.rennes.grid5000.fr.domain: 65121+ PTR? 5.98.16.172.in-addr.arpa. (42) 14:27:43.921384 IP dns.rennes.grid5000.fr.domain > paravance-60.rennes.grid5000.fr.38784: 65121* 1/1/0 PTR parapide-5.rennes.grid5000.fr. (103) 14:27:43.921510 IP paravance-60.rennes.grid5000.fr.49250 > dns.rennes.grid5000.fr.domain: 48890+ PTR? 111.111.16.172.in-addr.arpa. (45) 14:27:43.921816 IP dns.rennes.grid5000.fr.domain > paravance-60.rennes.grid5000.fr.49250: 48890* 1/1/0 PTR kadeploy.rennes.grid5000.fr. (104) 14:27:44.017208 ARP, Request who-has parapide-5.rennes.grid5000.fr tell dns.rennes.grid5000.fr, length 46 14:27:44.201278 IP6 fe80::214:4fff:feca:9470 > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28 14:27:44.201416 IP paravance-60.rennes.grid5000.fr.34416 > dns.rennes.grid5000.fr.domain: 7912+ PTR? 6.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.f.f.ip6.arpa. (90) 14:27:44.284641 ARP, Request who-has parapide-9.rennes.grid5000.fr tell kadeploy.rennes.grid5000.fr, length 46 14:27:44.307171 ARP, Request who-has parapide-5.rennes.grid5000.fr tell metroflux.rennes.grid5000.fr, length 46 14:27:44.398978 IP dns.rennes.grid5000.fr.domain > paravance-60.rennes.grid5000.fr.34416: 7912 NXDomain 0/1/0 (160)
We can see ARP requests, DNS messages, multicast reports…
Communication without routing : OpenVSwitch
Previously we already installed OpenVSwitch on our "gateway" node (node2). Now we will use it!
Our goal is to allow 2 nodes from 2 different VLANs to communicate without routing. To do this, nodes from each side must be in the same IP network, so change the IP of node_nancy to an IP in the local VLAN's subnet, but different from node1's and node2's IPs (192.168.192.0/20 in our situation => 192.168.200.2 for example). Then restart the networking service.
Now we have to set up the OpenVSwitch config on node2.
![]() |
Warning |
---|---|
node2 will lose its IP address, so the SSH session will be lost, use Kaconsole instead for this part |
- Create the bridge
- Remove IPs on both interfaces eth0 and eth1:
Add eth0 and eth1 to the bridge:
Now that it is done, you should be able to ping node_nancy (with its new IP!). If you run traceroute, you will notice that there is only one hop between node1 and node_nancy.
You can use openVSwitch to manage flows, for example you can DROP all packets from an IP with this command:
"in_port=1" refers to eth0 in our case. You can list all the ports in the bridge with:
You can also DROP all pings to a specific IP from a specific port:
And the following command will display all your flow rules :
![]() |
Note |
---|---|
If you want to know more about flow syntax go to this man page and look for the "Flow Syntax" paragraph |
Topo_maker
Topo_maker is a totally new tool on Grid'5000 (consider this as beta testing for the moment) it allows you to do all step of the part "set up topology" (except routing). There is no official documentations yet, but we will start from an example to do the same topology as above.
topo_maker wait a Rspec XML file as input, it is inspired by the syntax used in CloudLab (another Cloud platform), this is the example we will use :
<?xml version="1.0" encoding="UTF-8"?> <rspec> <node client_id="node-2"> <interface client_id="interface-0"/> <!--No IP precised equals to DHCP--> <interface client_id="interface-1"> <ip address="192.168.1.3" type="ipv4" netmask="255.255.255.0" /> </interface> <sliver_type name="raw-pc"> <disk_image name="jessie-x64-min" /> <!--This is just for the example because default OS chosen by topomaker is jessie-x64-min --> </sliver_type> <auto_install> <apt name="tcpdump"/> <apt name="openvswitch-switch"/> </auto_install> </node> <node client_id="node-1"> <!--Here we will have DHCP on jessie x64 min--> <interface client_id="interface-2"/> </node> <node client_id="node-3"> <!--node 3 is equivalent to our node_nancy here it will be on the same site, as we saw it doesn't matter with global vlan--> <interface client_id="interface-4"> <ip address="192.168.1.1" type="ipv4" netmask="255.255.255.0" /> </interface> </node> <link client_id="link-0"> <!--link are equivalent to VLAN--> <interface_ref client_id="interface-0" /> <interface_ref client_id="interface-2" /> </link> <link client_id="link-1"> <interface_ref client_id="interface-1" /> <interface_ref client_id="interface-4" /> </link> </rspec>
As you can see each part of this XML file is totally independant of the nodes you reserved, you can put the name of your choice for nodes name and interface name. It just have to be coherent between names in interfaces and names in links.
TopoMaker isn't released yet, so we will download the source code and use it directly.
topo_maker is in ruby and some gem aren't installed (yet).
Topo_maker doesn't sub your reservation you should have a reservation and its job id. In the reservation prompt you can exec "echo $OAR_JOBID" else you can find your reservation in "oarstat -u".
Now, we will start topo_maker :
You could add "-v" option to show more information about the progress and avoid to think the script is stuck. At the end a yaml file is prompt with all informations needed like IP or hostname. Topo_maker doesn't know about DNS in kavlan-local (and IP aren't get back) so the rules is the same as in the TP if you used DHCP : "hostname-kavlan-ID.rennes.grid5000.fr" from the gateway to connect to the node.
One the script is done, you can do the same test as above, and do yourself the modification to use openVSwitch with TopoMaker to exercise yourself.