Network reconfiguration tutorial
Note | |
---|---|
This page is actively maintained by the Grid'5000 team. If you encounter problems, please report them (see the Support page). Additionally, as it is a wiki page, you are free to make minor corrections yourself if needed. If you would like to suggest a more fundamental change, please contact the Grid'5000 team. |
Introduction
This TP aims to discover a method to configure a network in Grid'5000 using KaVLAN.
KaVLAN is a tool on Grid'5000 which allow to the user to manage VLANs in the platform. It edits switch configuration to change the VLAN number of the port corresponding to the interface of a node. This method permits complete layer 2 isolation.
Three kinds of VLANs are available on Grid'5000, you can find more information on the page KaVLAN. In this TP, we will use only global Vlan and local Vlan.
In the first time, we will set up a simple topology with 2 VLANs, a global and a local. In each VLAN there will be at least one node, and one interface of a node with 2 interfaces.
Set up topology
Reservations
A global VLAN is all over Grid'5000, so we have to reserve it on only one site. We will use Rennes and Nancy sites.
rennes:frontend :
|
oarsub -l {"type='kavlan-global'"}/vlan=1+{"type='kavlan-local'"}/vlan=1+{"cluster='paravance'"}/nodes=3,walltime=3 -I -t deploy |
With this reservation we have, 1 kavlan-local, 1 kavlan-global and 3 nodes on paravance cluster. Paravance is a cluster of nodes with 2 interfaces.
Get your Vlans ID :
How to know which VLAN is global and which is local ? It's simple, it's write in KaVLAN (look at the first diagram ;)) :
- kavlan-local : [1-3]
- kavlan : [4-9]
- global : [10-20]
Deployment
Now we will deploy our nodes with debian jessie minimal :
During the deployment you can reserve and deploy a node on Nancy in other terminal :
We will install 'at' and 'tcpdump' on each node using TakTuk :
rennes:node :
|
taktuk -s -l root -f $OAR_FILE_NODES broadcast exec [ "apt-get update; apt-get --yes install at tcpdump" ]
|
Since jessie, the default sshd configuration doesn't allow password authentication, then you can't connect to a deployed node from an other node. There is two solution
kaconsole
Kaconsole is a tool provided into Grid'5000, basically it allows to connect to a node like if you connect a screen and a keyboard on it.
So, you can connect with root:grid5000 on the node.
Add SSH key pair on each node
You can just use this small script from the reservation prompt :
#! /bin/bash temp=$(mktemp -d XXXXX) ssh-keygen -t rsa -f $temp/id_rsa -P "" for i in $(uniq $OAR_NODEFILE) ; do scp $temp/id_rsa root@$i:.ssh/id_rsa && ssh-copy-id -i $temp/id_rsa.pub root@$i done rm -r $temp
It will generate a temporary rsa pair of key, copy the private and allow the public on each node.
Network configuration
Nancy
We will put static IP on our node on nancy and put it in Vlan. So set this configuration on the first interface (we will assume this is eth0) in /etc/network/interfaces
auto eth0 iface eth0 inet static address 192.168.1.1 netmask 255.255.255.0
nancy:frontend :
|
ssh root@Hostname_node_nancy "apt-get --yes install at && echo 'service networking restart' | at now + 1 minute" && kavlan -s -i Global_Vlan_Id -m Hostname_Node_Nancy |
Rennes
As we have 3 nodes on Rennes, we will call them node1, node2 and node3, of course during the TP node1=the hostname of your first node. For example node1=paravance-23.
We will put the node1 in the local vlan and restart the networking service to get a new IP address (there is DHCP server in kavlan-local).
rennes:frontend :
|
ssh root@node1 "echo 'service networking restart' | at now + 1 minute" && kavlan -s -i Local_Vlan_Id -m node1 |
Warning | |
---|---|
Later we will configure an OpenVSwitch on node2, so, execute 'apt-get install openvswitch-switch' before putting node2 in another vlan than (production) DEFAULT vlan |
Now we will put a node between the local and global vlan using node2. We have to set up the second interface. We know eth1 is linked with the API : https://api.grid5000.fr/sid/sites/rennes/clusters/paravance/nodes/paravance-1.json?pretty
So we have to add to /etc/network/interfaces on node2 :
auto eth1 iface eth1 inet static address 192.168.1.2 netmask 255.255.255.0
We need to put the first interface in the local-vlan and the second in the global :
rennes:frontend :
|
ssh root@node2 "echo 'service networking restart' | at now + 1 minute" && kavlan -s -i Local_Vlan_Id -m node1 && kavlan -s -i Global_Vlan_Id -m node2-eth1 |
Now we have :
[prod <-]--kavlan-ID--[-> local-vlan] [local-vlan <--node1-->] [local-vlan <-]--node2--[-> global-vlan] [global-vlan <--node_nancy-->]
Now you should be able to ssh connect node2 from kavlan gateway, and ping 192.168.1.1 (node_nancy).
Routing
node1 and node_nancy are in two different VLAN, so currently it's impossible to send a packet between node1 and node_nancy. To do this, we will set route between the network in the local vlan and the network in the global VLAN (192.168.1.0/24)
Note | |
---|---|
To know the network address in the local vlan, connect to node1, node2 or kavlan-ID and enter the command : ip route |
Packets can't change of network without router between the vlans, so we will use our gateway (node2) to do this. Firstly we have to allow ip forwarding on node2.
Now, if a IP packet with known network destination is sent to node2, it will be forwarded to the destination network.
But node1 don't know how to reach node_nancy network and vice versa. We will add a route to each node.
Topology test
We will check if packet pass through our gateway :
Open two terminal :
On the first terminal you should see the ICMP packet are forwarded on the node2 :
IP 192.168.200.7 > 192.168.1.1: ICMP echo request, id 4270, seq 1, length 64 IP 192.168.1.1 > 192.168.200.7: ICMP echo reply, id 4270, seq 1, length 64 IP 192.168.200.7 > 192.168.1.1: ICMP echo request, id 4270, seq 2, length 64 IP 192.168.1.1 > 192.168.200.7: ICMP echo reply, id 4270, seq 2, length 64 IP 192.168.200.7 > 192.168.1.1: ICMP echo request, id 4270, seq 3, length 64 IP 192.168.1.1 > 192.168.200.7: ICMP echo reply, id 4270, seq 3, length 64
If you disable ip forwarding on node2, nothing will happen in tcpdump and ping will fail !
We can check the route with traceroute from node1 (or from node_nancy whatever) :
And the result is :
traceroute to 192.168.1.1 (192.168.1.1), 30 hops max, 60 byte packets 1 192.168.200.8 (192.168.200.8) 0.136 ms 0.122 ms 0.115 ms 2 192.168.1.1 (192.168.1.1) 25.612 ms 25.617 ms 25.611 ms
The first jump is from node1 to node2, and the second node is from node2 to node_nancy.
With tcpdump we can check the complete isolation of nodes from the production Vlan (or any other) :
On my node1 (in 12 seconds):
14:26:15.107927 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 82bd.8c:60:4f:47:6c:bc.808e, length 43 14:26:17.109436 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 82bd.8c:60:4f:47:6c:bc.808e, length 43 14:26:19.108669 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 82bd.8c:60:4f:47:6c:bc.808e, length 43 14:26:21.108675 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 82bd.8c:60:4f:47:6c:bc.808e, length 43 14:26:23.108669 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 82bd.8c:60:4f:47:6c:bc.808e, length 43 14:26:25.108654 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 82bd.8c:60:4f:47:6c:bc.808e, length 43 14:26:27.108674 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 82bd.8c:60:4f:47:6c:bc.808e, length 43
We just have spanning tree frame from the switch.
On a node in production vlan (in only one second) :
14:27:43.920934 IP paravance-60.rennes.grid5000.fr.38784 > dns.rennes.grid5000.fr.domain: 65121+ PTR? 5.98.16.172.in-addr.arpa. (42) 14:27:43.921384 IP dns.rennes.grid5000.fr.domain > paravance-60.rennes.grid5000.fr.38784: 65121* 1/1/0 PTR parapide-5.rennes.grid5000.fr. (103) 14:27:43.921510 IP paravance-60.rennes.grid5000.fr.49250 > dns.rennes.grid5000.fr.domain: 48890+ PTR? 111.111.16.172.in-addr.arpa. (45) 14:27:43.921816 IP dns.rennes.grid5000.fr.domain > paravance-60.rennes.grid5000.fr.49250: 48890* 1/1/0 PTR kadeploy.rennes.grid5000.fr. (104) 14:27:44.017208 ARP, Request who-has parapide-5.rennes.grid5000.fr tell dns.rennes.grid5000.fr, length 46 14:27:44.201278 IP6 fe80::214:4fff:feca:9470 > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28 14:27:44.201416 IP paravance-60.rennes.grid5000.fr.34416 > dns.rennes.grid5000.fr.domain: 7912+ PTR? 6.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.f.f.ip6.arpa. (90) 14:27:44.284641 ARP, Request who-has parapide-9.rennes.grid5000.fr tell kadeploy.rennes.grid5000.fr, length 46 14:27:44.307171 ARP, Request who-has parapide-5.rennes.grid5000.fr tell metroflux.rennes.grid5000.fr, length 46 14:27:44.398978 IP dns.rennes.grid5000.fr.domain > paravance-60.rennes.grid5000.fr.34416: 7912 NXDomain 0/1/0 (160)
We have ARP request, DNS message, multicast report, ...
Communication without routing : OpenVSwitch
Previously we already installed openswitch on our "gateway" node (node2). We will use it !
Our objective is to allow 2 nodes from 2 different VLANs to communicate without routing. To do this, nodes from each side must be in the same network, so change the IP configuration of node_nancy by something in agreement with the subnet of the vlan local but different of node1 and node 2(192.168.192.0/20 in our situation => 192.168.200.2 for example) and restart networking service.
Now we have to set up the OpenVSwitch config on node2.
- Create the bridge
- Delete IP from eth0 and eth1
Add interfaces eth0 and eth1 to the bridge :
It's done, you should be able to ping node_nancy (with its new IP !). If you do a traceroute, you will notice that : There is only one jump : node1 => node_nancy.