Network reconfiguration tutorial: Difference between revisions
Line 20: | Line 20: | ||
= Topology setup = | = Topology setup = | ||
In this first part, we setup a topology, as presented in the picture on the right [[File:TP-Kavlan-School2016.png|right|500px|thumb|Network topology to setup]] | |||
== Reservations of the resources== | == Reservations of the resources== | ||
Line 250: | Line 250: | ||
{{Term|location=rennes:frontend|cmd=<code class="command">ssh</code> root@<code class="replace">node2</code> "echo 'service networking restart' | at now + 1 minute" && kavlan -s -i <code class="replace">Local_Vlan_Id</code> -m <code class="replace">node1</code> && kavlan -s -i <code class="replace">Global_Vlan_Id</code> -m <code class="replace">node2-eth1</code>}} | {{Term|location=rennes:frontend|cmd=<code class="command">ssh</code> root@<code class="replace">node2</code> "echo 'service networking restart' | at now + 1 minute" && kavlan -s -i <code class="replace">Local_Vlan_Id</code> -m <code class="replace">node1</code> && kavlan -s -i <code class="replace">Global_Vlan_Id</code> -m <code class="replace">node2-eth1</code>}} | ||
== First | == First tests of the topology == | ||
We should now have | We should now have the topology described in the figure which is presented at the top of the page. | ||
Let's try some basic tests: | |||
# Try to ssh to the SSH gateway of the ''local VLAN'', then to <code class="replace">rennes-1</code>, then to <code class="replace">rennes-2</code> and finally to <code class="replace">nancy-1</code> | |||
# Try to ping <code class="replace">rennes-2</code> and then <code class="replace">rennes-1</code> from <code class="replace">nancy-1</code>. | |||
Only the last ping between <code class="replace">nancy-1</code> and <code class="replace">rennes-2</code> should fail. | |||
== Setting up the routing between the VLANs == | == Setting up the routing between the VLANs == |
Revision as of 19:38, 20 January 2016
![]() |
Note |
---|---|
This page is actively maintained by the Grid'5000 team. If you encounter problems, please report them (see the Support page). Additionally, as it is a wiki page, you are free to make minor corrections yourself if needed. If you would like to suggest a more fundamental change, please contact the Grid'5000 team. |
Introduction
The goal of this tutorial is to present a example of usage of Grid'5000, which configure a customized network topology for an experiment, using KaVLAN.
KaVLAN is a tool of Grid'5000, which allows a user to manage VLANs in the platform. Behind the scene, KaVLAN actually changes the configuration of the network switchs, to set the VLAN membership (VLAN ID) for the ports which corresponds to the network interface of one of more nodes. That mechanism provides a layer 2 network isolation for experiments.
Three kinds of KaVLAN VLANs are available on Grid'5000. You can find more information in the KaVLAN page. In this tutorial, we will only use global and local VLANs (but no routed VLAN).
Before going further with the practical setup proposed in this page, it is strongly advised the read the KaVLAN page.
Topology setup
In this first part, we setup a topology, as presented in the picture on the right
Reservations of the resources
We will now create jobs to reserve:
- 2 nodes providing at least 2 network interfaces each, on one site
- 1 local VLAN on the same site
- 1 global VLAN on the same site (but any site would be fine)
- 1 node on another site
In the text below, we propose to use the sites of Rennes and Nancy:
- Rennes for the 2 nodes with several network interfaces (but Lyon and Nancy also provide such nodes, see Special_Features), the local VLAN, and the global VLAN (even if, again, the site where you reserve the global VLAN doesn't matter in this tutorial)
- Nancy for the other node
Consequently, practically speaking, you are advised to open 2 terminal windows (2 separate shells), one for Rennes and one for Nancy, for the tutorial.
In Rennes (Rennes terminal)
In Rennes, we have to reserve 1 local VLAN, and 2 nodes for instance of the paravance cluster, because that cluster is composed of nodes with 2 network interfaces.
We can also try to reserve the global VLAN in Rennes as well.
![]() |
Note |
---|---|
Please mind the fact that a global VLAN spreads all over Grid'5000, so we only need to reserve it on one site. |
Since we have to work on 2 different sites, we will use OAR advance reservations in order to have 2 jobs that start and end at the same dates. Let's define STARTDATE
as the start date (e.g. "2016-02-04 14:00:00", don't worry if it is in the past for a few minutes), and WALLTIME
as the duration of the job (e.g. the duration for the tutorial).
Once logged to Rennes' frontend in our first terminal, and we execute:
![]() |
rennes:frontend :
|
oarsub -t deploy -l {"type='kavlan-global'"}/vlan=1+{"type='kavlan-local'"}/vlan=1+{"cluster='paravance'"}/nodes=3,walltime=WALLTIME -r STARTDATE |
Unfortunately, that command may fail (return KO):
- either because there are no more paravance nodes available for the time of the tutorial, or no more local VLAN: that should not be too likely to happen
- or because the global VLAN of Rennes is already taken: that is very likely to happen during a common tutorial session with many students !
In either case, one will have to try to use another site than Rennes, or try again later.
Let say we only meet issue #2: in that case, we will only reserve 1 local VLAN, and 2 nodes for instance of the paravance cluster in Rennes, but keep the global VLAN reservation for the other site.
For the simplicity of writing of this tutorial, we assume that the global VLAN of Nancy is available (but any other site with a free global VLAN could do).
So, unless the previous command succeed, we execute:
![]() |
rennes:frontend :
|
oarsub -t deploy -l {"type='kavlan-local'"}/vlan=1+{"cluster='paravance'"}/nodes=3,walltime=WALLTIME -r "STARTDATE " |
Again, for the simplicity of this writing, let's call our 2 nodes in Rennes rennes1
and rennes-2
(e.g. you might have rennes1
= paravance-23).
Either way, a successful oarsub
command should have returned a JOBID
.
Now, let's retrieve the ID of our VLANs (local + global, or just the local one if the global VLAN is reserved elsewhere):
You can know which VLAN ID is global and which is local, using the following table:
KaVLAN name in OAR | type | first id | last id |
---|---|---|---|
kavlan-local | local | 1 | 3 |
kavlan | routed | 4 | 9 |
kavlan-global | global | 10 | 21 |
(see KaVLAN for more details)
In Nancy (Nancy terminal)
If we succeed to reserve the global VLAN in Rennes, we just need to reserve a single node in Nancy. In our second terminal, we log to Nancy's frontend, and execute:
If not, we also reserve Nancy's global VLAN. We execute:
![]() |
nancy:frontend :
|
oarsub -t deploy -l {"type='kavlan-global'"}/vlan=1+nodes=1,walltime=WALLTIME -r "STARTDATE " |
Again, a successful oarsub
command should have returned a JOBID
(which will of course be different than from the job in Rennes).
If the oarsub
fails (reservation is said "KO"), that means that you probably should try to use yet another site than Nancy, where the global VLAN would be available.
For the simplicity of this writing, let's call our node in Nancy nancy-1
(e.g. nancy-1
= graphite-3).
You can now look at which VLANs you get, as above.
Deployment of our operating system for the experiment
In Rennes (Rennes terminal)
Back on the Rennes termnal, we now enter our job in order to benefit from OAR environment variable $OAR_NODEFILE.
(JOBID
is the one of the job of Rennes of course)
Then we launch the deployement of Debian jessie minimal environment on the nodes:
In Debian jessie image, the default sshd configuration doesn't allow password authentication, as a result you cannot connect from a deployed node to another node via password. However, there are several solutions to overcome that issue:
- kaconsole
Kaconsole is a tool provided in Grid'5000, basically it allows to connect to the serial console of a node, quite the same way you would connect to a virtual console of a GNU/Linux workstation (CTRL+ALT+F1).
Credentials for login on the console are: "root":"grid5000".
- Add a SSH key pair on every node
Even though it is very useful in last resort, using the console is not very convenient. So we will actually push SSH keys on the nodes.
You can use the following bash commands from the reservation shell (where you did the oarsub
), to generate and copy a SSH key pair in the home directory of the root user on your nodes:
#! /bin/bash temp=$(mktemp -d XXXXX) ssh-keygen -t rsa -f $temp/id_rsa -P "" for i in $(uniq $OAR_NODEFILE) ; do scp $temp/id_rsa root@$i:.ssh/ && ssh-copy-id -i $temp/id_rsa.pub root@$i done rm -r $temp
It will generate a temporary RSA key pair, copy the private key on each node, and finally authorize the connection using that key.
In Nancy (Nancy terminal)
We run from Nancy's frontend the same commands as in Rennes, but use the JOBID
of Nancy of course.
Installing useful extra packages for networking
In this tutorial we are going to need some extra packages which are not installed in jessie-min environment. And as we are going to modify the network configurations making the Internet unreachable from the nodes, we must install them beforehand.
Besides, you may wish to play with (or debug) your network configuration with useful tools like dig, traceroute, ethtool or ipcalc, which are not in jessie-min either. If you want to use them later you need to install them now too.
In Nancy (Nancy terminal)
![]() |
nancy:frontend :
|
ssh root@nancy-1 "apt-get update; apt-get install --yes at tcpdump dnsutils traceroute ethtool ipcalc" |
In Rennes (Rennes terminal)
In Rennes, since we reserved several nodes, we can use TakTuk for speeding up the process:
![]() |
rennes:frontend :
|
taktuk -s -l root -f $OAR_FILE_NODES broadcast exec [ "apt-get update; apt-get --yes install at tcpdump dnsutils traceroute ethtool ipcalc" ] |
In addition, we will need OpenVSwitch on node-2 in the second part of the tutorial:
Network configuration
In Nancy (Nancy terminal)
We will put a static IP on nancy-1, and put it in the global VLAN we reserved. To do so, we will configure the network interface.
The question is: what IP will we give to the node?
If we set an arbitrary IP in an random IP range, the node won't be reachable anymore because we will try to contact it via the site router (default route), and the router won't know either where to send these packets. To address this problem, Grid'5000 is configured with IP ranges dedicated to each VLAN. You can check these ranges in the Golden Rules page.
(bpichot) ^ am I mistaking here?
Besides, there is a naming issue with giving an arbitrary IP to a node, because the DNS will obviously not be aware of this IP change. So basically we won't be able to run commands using the name nancy-1
. Fortunately, there are static DNS entries in Grid'5000 for each node in each VLAN, to be always able to call nodes with a name instead of an IP. These names are based on this pattern: node_name
-kavlan-VLAN_ID
. For instance, the built-in name for the node graphite-3 in the global VLAN 16 is: "graphite-3-kavlan-16".
Consequently, it will be much more convenient to configure nancy-1
with the IP already associated to its name in the DNS.
This IP will be given by the 'host' command:
Example:
graphite-3-kavlan-16.nancy.grid5000.fr has address 10.27.200.239
So let's configure the interface with this IP (we assume here that is is eth0, but you can check what interface is in use on the node with the "ip address" command), and the "/18" netmask mentioned in the Golden Rules page.
auto eth0 iface eth0 inet static address <nancy-1's IP in global VLAN> netmask 255.255.192.0
![]() |
Note |
---|---|
To set directly a new IP on your node while it is still in the production VLAN is generally a bad idea, because you could create conflicts of addresses with other parts of the network. |
So, we exit the node to put it in the global VLAN before applying the new network configuration. Let's try to do so with a naive approach:
Now, we want to apply the new configuration on nancy-1
:
But this command should timeout! It makes sense, because the node is not available in the frontend's network anymore.
To put nancy-1
back in the production VLAN, do the following on the frontend:
And of course, the opposite way would fail…
(bpichot) ^ uhh… either way, once in the global VLAN it will be unreachable unless using kaconsole3, since the global VLAN is not routed :/
The following command, based on the 'at' command, is a better approach, in order to reconfigure the network after setting the VLAN.
![]() |
nancy:frontend :
|
ssh root@nancy-1 "echo '(service networking restart ; killall dhclient)' | at now + 1 minute" && kavlan -s -i Global_Vlan_Id -m nancy-1 .nancy.grid5000.fr --verbose |
In Rennes (Rennes terminal)
We need to put node1 in the local VLAN and restart the networking service to get a new IP address thanks to the DHCP service (which is provided in every kavlan-local VLAN).
Like earlier, we need the 'at' command to do that.
![]() |
rennes:frontend :
|
ssh root@node1 "echo 'service networking restart' | at now + 1 minute" && kavlan -s -i Local_Vlan_Id -m rennes-1 .rennes.grid5000.fr |
![]() |
Warning |
---|---|
Be careful not to mix up the syntax of |
Before changing rennes-2
's network configuration, and since we will need to have OpenVSwitch installed on that node in the second part of this tutorial, we first install the openvswitch-switch package before the node leaves the production VLAN, as it won't be able to access the Internet afterwards :
We can now put rennes-2 in between the local and global VLANs: the first interface will be on the local VLAN, while the second one will be on the global VLAN.
Like above, the first interface, which is in the local VLAN, will get an IP from the DHCP service which is provided here. But we have to setup a static IP configuration for the second interface: eth1. (according to the information of the Grid'5000 API, eth1 is indeed cabled to the switch: see eth1 in https://api.grid5000.fr/sid/sites/rennes/clusters/paravance/nodes/paravance-1.json?pretty).
For that purpose we need to edit /etc/network/interfaces on node2:
auto eth1 iface eth1 inet static address 192.168.1.2 netmask 255.255.255.0
We new switch the first interface to the local VLAN and the second interface in the global VLAN:
![]() |
rennes:frontend :
|
ssh root@node2 "echo 'service networking restart' | at now + 1 minute" && kavlan -s -i Local_Vlan_Id -m node1 && kavlan -s -i Global_Vlan_Id -m node2-eth1 |
First tests of the topology
We should now have the topology described in the figure which is presented at the top of the page.
Let's try some basic tests:
- Try to ssh to the SSH gateway of the local VLAN, then to
rennes-1
, then torennes-2
and finally tonancy-1
- Try to ping
rennes-2
and thenrennes-1
fromnancy-1
.
Only the last ping between nancy-1
and rennes-2
should fail.
Setting up the routing between the VLANs
node1 and node_nancy are in two different VLANs (node1 is in a local VLAN, while node_nancy is in a global VLAN). As a result, it is for now impossible to send a packet between node1 and node_nancy.
To enable that, we will setup a route between the local VLAN and the global VLAN (192.168.1.0/24).
Since node2 has 2 network interfaces, with one in each of the VLANs, we will configure it to act as a router (also called gateway sometime).
![]() |
Note |
---|---|
In order to know the IP network subnet used in the local VLAN, we can connect to node1, node2 or kavlan-ID and enter the command : ip route |
![]() |
Warning |
---|---|
For the ease of writing of this tutorial, we will assume here that that network is 192.168.192.0/20 |
First, we have to enable routing on node2:
Now, if an IP packet with a destination in a known network is sent to node2, it will be forwarded.
But node1 deosn't know how to reach node_nancy's network, and vice versa. Let's add a route to each node:
Testing
We first check if packets are able to find their way via our router.
We run on node1 and node2 (in Rennes) the following commands:
(tcpdump is a network traffic sniffer, here filtering ICMP traffic on eth0)
On the first terminal you should see that the ICMP packets are forwarded on node2 :
IP 192.168.200.7 > 192.168.1.1: ICMP echo request, id 4270, seq 1, length 64 IP 192.168.1.1 > 192.168.200.7: ICMP echo reply, id 4270, seq 1, length 64 IP 192.168.200.7 > 192.168.1.1: ICMP echo request, id 4270, seq 2, length 64 IP 192.168.1.1 > 192.168.200.7: ICMP echo reply, id 4270, seq 2, length 64 IP 192.168.200.7 > 192.168.1.1: ICMP echo request, id 4270, seq 3, length 64 IP 192.168.1.1 > 192.168.200.7: ICMP echo reply, id 4270, seq 3, length 64
If ip forwarding was disabled on node2 (see above the sysctl command to disable it), nothing will happen in tcpdump and ping should report errors.
We also check the packet route with traceroute from node1:
We should see two hops: the intermediary router and the target:
traceroute to 192.168.1.1 (192.168.1.1), 30 hops max, 60 byte packets 1 192.168.200.8 (192.168.200.8) 0.136 ms 0.122 ms 0.115 ms 2 192.168.1.1 (192.168.1.1) 25.612 ms 25.617 ms 25.611 ms
The first hop is from node1 to node2, and the second one is from node2 to node_nancy.
With tcpdump we can check the complete isolation of nodes from the production VLAN (and from any other one) :
On node1 (in a 12 seconds time):
14:26:15.107927 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 82bd.8c:60:4f:47:6c:bc.808e, length 43 14:26:17.109436 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 82bd.8c:60:4f:47:6c:bc.808e, length 43 14:26:19.108669 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 82bd.8c:60:4f:47:6c:bc.808e, length 43 14:26:21.108675 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 82bd.8c:60:4f:47:6c:bc.808e, length 43 14:26:23.108669 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 82bd.8c:60:4f:47:6c:bc.808e, length 43 14:26:25.108654 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 82bd.8c:60:4f:47:6c:bc.808e, length 43 14:26:27.108674 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 82bd.8c:60:4f:47:6c:bc.808e, length 43
The only frames received are spanning tree frames, from the switch.
In comparison, this is what can be captured on a node in the production VLAN, in only a one second time:
14:27:43.920934 IP paravance-60.rennes.grid5000.fr.38784 > dns.rennes.grid5000.fr.domain: 65121+ PTR? 5.98.16.172.in-addr.arpa. (42) 14:27:43.921384 IP dns.rennes.grid5000.fr.domain > paravance-60.rennes.grid5000.fr.38784: 65121* 1/1/0 PTR parapide-5.rennes.grid5000.fr. (103) 14:27:43.921510 IP paravance-60.rennes.grid5000.fr.49250 > dns.rennes.grid5000.fr.domain: 48890+ PTR? 111.111.16.172.in-addr.arpa. (45) 14:27:43.921816 IP dns.rennes.grid5000.fr.domain > paravance-60.rennes.grid5000.fr.49250: 48890* 1/1/0 PTR kadeploy.rennes.grid5000.fr. (104) 14:27:44.017208 ARP, Request who-has parapide-5.rennes.grid5000.fr tell dns.rennes.grid5000.fr, length 46 14:27:44.201278 IP6 fe80::214:4fff:feca:9470 > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28 14:27:44.201416 IP paravance-60.rennes.grid5000.fr.34416 > dns.rennes.grid5000.fr.domain: 7912+ PTR? 6.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.f.f.ip6.arpa. (90) 14:27:44.284641 ARP, Request who-has parapide-9.rennes.grid5000.fr tell kadeploy.rennes.grid5000.fr, length 46 14:27:44.307171 ARP, Request who-has parapide-5.rennes.grid5000.fr tell metroflux.rennes.grid5000.fr, length 46 14:27:44.398978 IP dns.rennes.grid5000.fr.domain > paravance-60.rennes.grid5000.fr.34416: 7912 NXDomain 0/1/0 (160)
Here we see ARP requests, DNS messages, multicast reports…
Communication without routing: OpenVSwitch
- Setup
Our goal is to allow 2 nodes from 2 different (physical) VLANs to communicate without routing. To do this, nodes from each side must be in the same IP network. For that, our two layer 2 VLANs must be bridged. In this tutorial, we will not use Linux bridge (the brctl
command), but OpenVSwitch, which provides enhanced functionalities.
First we need to change the IP of node_nancy to an IP in the local VLAN's subnet (but different from the IPs of node1 and node2 of course) (192.168.192.0/20 in our case, we can use 192.168.200.2 for instance). Then restart the networking service.
We already installed OpenVSwitch on our router node, node2, we can now use it.
![]() |
Warning |
---|---|
node2 will lose its IP address, so the SSH session will be lost, use Kaconsole instead for the following |
- Create the bridge
- Remove IPs on both interfaces eth0 and eth1:
Add eth0 and eth1 to the bridge:
- Tests
Now that it is done, you should be able to ping node_nancy (with its new IP!).
Also if you run traceroute, you will notice that there is only one hop between node1 and node_nancy.
- Flow control
You can use openVSwitch to manage flows, for example you can DROP all packets from an IP with this command:
"in_port=1" refers to eth0 in our case. You can list all the ports in the bridge with:
You can also DROP all pings to a specific IP from a specific port:
And the following command will display all your flow rules :
![]() |
Note |
---|---|
If you want to know more about flow syntax go to this man page and look for the "Flow Syntax" paragraph |
Automating the configuration with TopoMaker
TopoMaker is a totally new tool on Grid'5000 (software is still in beta testing for now), which allows to automate all steps of the "Topology setup" part (except the last paragraph about routing).
There is no documentations yet, but we will start from an example, to build the same topology as in the first paragraphs.
TopoMaker takes a Rspec XML file as input. That input is actually inspired by the syntax used in CloudLab (another experimental testbed). Here is the example we will use:
<?xml version="1.0" encoding="UTF-8"?> <rspec> <node client_id="node-2"> <interface client_id="interface-0"/> <!--No IP precised equals to DHCP--> <interface client_id="interface-1"> <ip address="192.168.1.3" type="ipv4" netmask="255.255.255.0" /> </interface> <sliver_type name="raw-pc"> <disk_image name="jessie-x64-min" /> <!--This is just for the example because default OS chosen by topomaker is jessie-x64-min --> </sliver_type> <auto_install> <apt name="tcpdump"/> <apt name="openvswitch-switch"/> </auto_install> </node> <node client_id="node-1"> <!--Here we will have DHCP on jessie x64 min--> <interface client_id="interface-2"/> </node> <node client_id="node-3"> <!--node 3 is equivalent to our node_nancy here it will be on the same site, as we saw it doesn't matter with global vlan--> <interface client_id="interface-4"> <ip address="192.168.1.1" type="ipv4" netmask="255.255.255.0" /> </interface> </node> <link client_id="link-0"> <!--link are equivalent to VLAN--> <interface_ref client_id="interface-0" /> <interface_ref client_id="interface-2" /> </link> <link client_id="link-1"> <interface_ref client_id="interface-1" /> <interface_ref client_id="interface-4" /> </link> </rspec>
As you can see, each part of this XML file is totally independent from the nodes you reserved. You can put the name of your choice for nodes and interfaces names. It just have to be relevant between names for interfaces and names for links.
TopoMaker isn't provided as a Grid'5000 tool yet, so we will download the source code and use it directly.
TopoMaker is developed in Ruby and requires some gems, which are not installed in the environement. We install them:
TopoMaker does not do the reservation for you. You should have a running job and provide its jobid.
The TopoMaker command line is as follows:
You can add the "-v" option to show more information about the progress of the execution (the script might look like stuck sometime). In the end, a yaml file is prompted with all the required information, like IP or hostname.
TopoMaker doesn't know about DNS in kavlan-local (and IP are not retrieved), so the rules is the same as in the tutorial if you used DHCP: "hostname-kavlan-ID.rennes.grid5000.fr" from the gateway to connect to the node.
Once the script is done, you can do the same tests as above, and do the modification yourself to use openVSwitch with TopoMaker.