Network reconfiguration tutorial: Difference between revisions
Line 170: | Line 170: | ||
{{Term|location=nancy:frontend|cmd=<code class="command">host</code> <code class="replace">nancy-1</code>-kavlan-<code class="replace">G_ID</code>}} | {{Term|location=nancy:frontend|cmd=<code class="command">host</code> <code class="replace">nancy-1</code>-kavlan-<code class="replace">G_ID</code>}} | ||
Since the IP configuration of the network | Since the IP configuration of the network Interface uses DHCP in the environnement we deployed, the mechanism we have to trigger is the following: | ||
# Change the VLAN of the node | # Change the VLAN of the node | ||
# Restart the network service | # Restart the network service |
Revision as of 00:24, 21 January 2016
Note | |
---|---|
This page is actively maintained by the Grid'5000 team. If you encounter problems, please report them (see the Support page). Additionally, as it is a wiki page, you are free to make minor corrections yourself if needed. If you would like to suggest a more fundamental change, please contact the Grid'5000 team. |
Introduction
The goal of this tutorial is to present a example of usage of Grid'5000, which benefits from the capability of reconfiguration of the network topology of the platform, using KaVLAN.
KaVLAN is a tool of Grid'5000, which allows a user to manage VLANs in the platform. Behind the scene, KaVLAN actually changes the configuration of the network switchs, to set the VLAN membership (VLAN ID) for the ports which are cabled to the network interfaces of one or more nodes. That mechanism can provide a layer 2 network isolation for experiments.
Three kinds of KaVLAN VLANs are available on Grid'5000. You can find more information in the KaVLAN page. In this tutorial, we will only use global and local VLANs (but no routed VLAN).
Before going further with the practical setup proposed in this page, it is strongly advised the read the KaVLAN page.
Topology setup
In this first part of the tutorial, we setup a topology, as presented in the following picture, except the router software, which is shown on node rennes-2
and will be setup in the second part.
Reservations of the resources
First of all we will create jobs to reserve:
- 2 nodes providing at least 2 network interfaces each, on one site
- 1 local VLAN on the same site
- 1 global VLAN
- 1 node on another site
In the text below, we use the sites of Rennes and Nancy (but other sites could be used, see below):
- Rennes for the 2 nodes with several network interfaces, and the local VLAN.
- Nancy for the other node
(we'll discuss later about the global VLAN).
Consequently, practically speaking, you are advised to open 2 terminal windows (2 separate shells) for the tutorial, one for Rennes and one for Nancy.
In Rennes (Rennes terminal)
In Rennes, we have to reserve 1 local VLAN, and 2 nodes for instance of the paravance cluster, because that cluster is composed of nodes with 2 network interfaces (see the Special Features page, for other Grid'5000 clusters providing such a characteristic).
We also try to reserve the global VLAN in Rennes as well.
Note | |
---|---|
Please mind the fact that a global VLAN spreads all over Grid'5000, so we only need to reserve it on one site. |
Since we have to work on 2 different sites, we will use OAR advance reservations in order to have 2 jobs that start and end at the same dates. Let's define STARTDATE
as the start date (e.g. "2016-02-04 14:00:00", don't worry if it is in the past for a few minutes), and WALLTIME
as the duration of the job (e.g. the duration for the tutorial).
Once logged to Rennes' frontend in our first terminal, we execute:
rennes:frontend :
|
oarsub -t deploy -l {"type='kavlan-global'"}/vlan=1+{"type='kavlan-local'"}/vlan=1+{"cluster='paravance'"}/nodes=3,walltime=WALLTIME -r STARTDATE |
Unfortunately, that command may fail (return KO):
- either because there are no more paravance nodes available for the time of the tutorial, or no more local VLAN: that should not be too likely to happen
- or because the global VLAN of Rennes is already taken: that is very likely to happen during a common tutorial session with many students !
In either case, one will have to try to use another site than Rennes, or try again later.
- Let say we only meet issue #2
In that case, we will only reserve 1 local VLAN, and 2 nodes of the paravance cluster, but keep the global VLAN reservation for the other site (for the simplicity of writing of this tutorial, we assume that the global VLAN of Nancy is available, but any other site with a free global VLAN could do).
So, unless the previous command succeed, we execute:
rennes:frontend :
|
oarsub -t deploy -l {"type='kavlan-local'"}/vlan=1+{"cluster='paravance'"}/nodes=3,walltime=WALLTIME -r "STARTDATE " |
For the simplicity of this writing, let's call our 2 nodes in Rennes rennes1
and rennes-2
(e.g. you might have rennes1
= paravance-23).
Either way, a successful oarsub
command should have returned a JOBID
.
Now, let's retrieve the ID of our VLANs (local + global, or just the local one if the global VLAN is reserved elsewhere):
You can know which VLAN ID is global and which is local, using the following table:
KaVLAN name in OAR | type | first id | last id |
---|---|---|---|
kavlan-local | local | 1 | 3 |
kavlan | routed | 4 | 9 |
kavlan-global | global | 10 | 21 |
(see KaVLAN for more details)
In Nancy (Nancy terminal)
If we succeed to reserve the global VLAN in Rennes, we just need to reserve a single node in Nancy. In our second terminal, we log to Nancy's frontend, and execute:
If not, we also reserve Nancy's global VLAN. We execute:
nancy:frontend :
|
oarsub -t deploy -l {"type='kavlan-global'"}/vlan=1+nodes=1,walltime=WALLTIME -r "STARTDATE " |
Again, a successful oarsub
command should have returned a JOBID
(which of course may certainly not be the same as the one of the job in Rennes).
If the oarsub
fails (reservation is said "KO"), that means that you probably should try to use yet another site than Nancy, where the global VLAN would be available.
For the simplicity of this writing, let's call our node in Nancy nancy-1
(e.g. nancy-1
= graphite-3).
You can now look at which VLANs you get, as above.
Deployment of our operating system for the experiment
In Rennes (Rennes terminal)
Back on the Rennes termnal, we now enter our job in order to benefit from OAR environment variable $OAR_NODEFILE.
(JOBID
is the one of the job of Rennes of course)
Then we launch the deployement of the Debian Jessie minimal environment to the nodes:
Note | |
---|---|
We choose to not use the --vlan |
In order to be capable of doing ssh
from one node to the other, we push our SSH private key to the nodes.
Note | |
---|---|
When access to a node is broken due to a error in a network setup for instance, the Kaconsole tool becomes very handy: it allows to connect to the serial console of a node, quite the same way you would connect to a virtual console of a GNU/Linux workstation (CTRL+ALT+F1), to for instance fix the network configuration. Credentials for login on the console with our deployed environment is "root":"grid5000". |
In Nancy (Nancy terminal)
We run from Nancy's frontend the same commands as in Rennes, but use the JOBID
of Nancy of course.
Installing useful extra packages for networking
In this tutorial we are going to need some extra packages which are not installed in jessie-min environment, such as tcpdump
. Since we are going to modify the network configurations making the Internet unreachable from the nodes, we must install them beforehand.
Besides, you may wish to play with (or debug) your network configuration with other tools like
ethtool
, dig
, or traceroute
as well. If you want to use them later you need to install them now too.
In Nancy (Nancy terminal)
In Rennes (Rennes terminal)
In Rennes, since we reserved several nodes (hey, 2 is more than 1 ! :-) ), we will use TakTuk for speeding up the process:
In addition, we will need OpenVSwitch on rennes-2
in the second part of the tutorial, we install it also:
Network configuration
Thanks to KaVLAN, we will have a layer 2 network isolation for our nodes. That means that whatever IP configuration we set in our VLAN, there will be no risk to break the network configuration in the default VLAN or to disturb other users (for instance by misusing reserved IP addresses). We can even setup network services or protocols based on IP broadcast for instance (e.g. DHCP) with no risk of polluting the default VLAN (or conflicting with the platform's defaut VLAN DHCP service).
However, we must keep in mind that:
- We must be very careful to what VLAN the node is in at any time. Indeed, a mistake could quickly lead to some changes to the nodes network configuration while the node is actually still in the default VLAN, which could impact other users/experiments.
- While it can be interesting to manage by ourselves the IP addressing of the nodes in our VLAN, Grid'5000 provides predefined IP addresses for within any VLAN, which can be very handy (see the KaVLAN page), especially because they are served by a DHCP service within each VLAN. So both options are possible, and one has to weigh the pros and the cons, depending on the actual need.
- Setting up our own IP addressing mechanism can actually be seen as an "expert mode", which the
kavlan
command provides some support to as well, allowing to disable the DHCP service in a VLANs is required.
- If not using the IP addressed which are provided by Grid'5000, routing to Global VLANs and Routed VLANs will not work, and neither will do SSH gateways to local VLANs. However, if all or some of those feature as not useful for your experiment, it is not harmful.
That said, for this tutorial and up until the openvswitch section, we will use the predefined IP addresses provided by Grid'5000. This will be very handy thanks to the DHCP services which are provided in the VLANs.
Furthermore, for the ease of reading of this document, let's have:
G_ID
= the ID of the global VLAN we reserved
L_ID
= the ID of the local VLAN we reserved
In Nancy (Nancy terminal)
The nancy-1
node is in the global VLAN. Thus, it will get in there the IP address which correspond to nancy-1
-kavlan-G_ID
. See the table in the KaVLAN page to find out the correponding IP address, or run:
Since the IP configuration of the network Interface uses DHCP in the environnement we deployed, the mechanism we have to trigger is the following:
- Change the VLAN of the node
- Restart the network service
Since out node won't be reachable with is old IP address (the one for the default VLAN) once put in the global VLAN, we have to create before action #1, a trigger for running action #2 asynchronously. That can be done using the at
for instance. Let's do that in one command line:
Another option would be to use Kaconsole, to access the node via its serial console to run action #2, but that is definitely not scalable to many nodes, so it is to be kept as a last resort option.
Feel free to bullet make different tests, with at
or with Kaconsole. To start over, you can put the node back in the default VLAN with the following command:
In Rennes (Rennes terminal)
We need to put node1 in the local VLAN and restart the networking service to get a new IP address thanks to the DHCP service (which is provided in every kavlan-local VLAN).
Like earlier, we need the 'at' command to do that.
Before changing rennes-2
's network configuration, and since we will need to have OpenVSwitch installed on that node in the second part of this tutorial, we first install the openvswitch-switch package before the node leaves the production VLAN, as it won't be able to access the Internet afterwards :
We can now put rennes-2 in between the local and global VLANs: the first interface will be on the local VLAN, while the second one will be on the global VLAN.
Like above, the first interface, which is in the local VLAN, will get an IP from the DHCP service which is provided here. But we have to setup a static IP configuration for the second interface: eth1.
(according to the information of the Grid'5000 API, eth1 is indeed cabled to the switch: see eth1 in https://api.grid5000.fr/sid/sites/rennes/clusters/paravance/nodes/paravance-1.json?pretty).
For that purpose we need to edit /etc/network/interfaces on node2:
auto eth1
iface eth1 inet static
address 192.168.1.2
netmask 255.255.255.0
We new switch the first interface to the local VLAN and the second interface in the global VLAN:
First tests of the topology
We should now have the topology described in the figure which is presented at the top of the page.
Let's try some basic tests:
- Try to ssh to
nancy-1
in the global VLAN from the frontend of any Grid'5000 site.
- Try to ssh to the SSH gateway of the local VLAN, then to
rennes-1
, then to rennes-2
and finally to nancy-1
- Try to ping
rennes-2
IP in itslocal VLAN, then in the global VLAN from nancy-1
- Try to ping
rennes-1
IP in thelocal VLAN from nancy-1
- Try to ping
nancy-1
IP in theglobal VLAN from rennes-1
For that purpose, you will have to find out the hostnames and IP addresses of the machines in the VLANs ! See KaVLAN.
Only the last pings between nancy-1
and rennes-2
(either way) should fail.
Setting up the IP routing between the VLANs
node1 and node_nancy are in two different VLANs (node1 is in a local VLAN, while node_nancy is in a global VLAN).
As a result, it is for now impossible to send a packet between node1 and node_nancy.
To enable that, we will setup a route between the local VLAN and the global VLAN (192.168.1.0/24).
Since node2 has 2 network interfaces, with one in each of the VLANs, we will configure it to act as a router (also called gateway sometime).
First, we have to enable routing on node2:
Now, if an IP packet with a destination in a known network is sent to node2, it will be forwarded.
But node1 deosn't know how to reach node_nancy's network, and vice versa. Let's add a route to each node:
New tests with IP routing enabled
We first check if packets are able to find their way via our router.
We run on node1 and node2 (in Rennes) the following commands:
(tcpdump is a network traffic sniffer, here filtering ICMP traffic on eth0)
On the first terminal you should see that the ICMP packets are forwarded on node2 :
IP 192.168.200.7 > 192.168.1.1: ICMP echo request, id 4270, seq 1, length 64
IP 192.168.1.1 > 192.168.200.7: ICMP echo reply, id 4270, seq 1, length 64
IP 192.168.200.7 > 192.168.1.1: ICMP echo request, id 4270, seq 2, length 64
IP 192.168.1.1 > 192.168.200.7: ICMP echo reply, id 4270, seq 2, length 64
IP 192.168.200.7 > 192.168.1.1: ICMP echo request, id 4270, seq 3, length 64
IP 192.168.1.1 > 192.168.200.7: ICMP echo reply, id 4270, seq 3, length 64
If ip forwarding was disabled on node2 (see above the sysctl command to disable it), nothing will happen in tcpdump and ping should report errors.
We also check the packet route with traceroute from node1:
We should see two hops: the intermediary router and the target:
traceroute to 192.168.1.1 (192.168.1.1), 30 hops max, 60 byte packets
1 192.168.200.8 (192.168.200.8) 0.136 ms 0.122 ms 0.115 ms
2 192.168.1.1 (192.168.1.1) 25.612 ms 25.617 ms 25.611 ms
The first hop is from node1 to node2, and the second one is from node2 to node_nancy.
With tcpdump we can check the complete isolation of nodes from the production VLAN (and from any other one) :
On node1 (in a 12 seconds time):
14:26:15.107927 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 82bd.8c:60:4f:47:6c:bc.808e, length 43
14:26:17.109436 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 82bd.8c:60:4f:47:6c:bc.808e, length 43
14:26:19.108669 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 82bd.8c:60:4f:47:6c:bc.808e, length 43
14:26:21.108675 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 82bd.8c:60:4f:47:6c:bc.808e, length 43
14:26:23.108669 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 82bd.8c:60:4f:47:6c:bc.808e, length 43
14:26:25.108654 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 82bd.8c:60:4f:47:6c:bc.808e, length 43
14:26:27.108674 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 82bd.8c:60:4f:47:6c:bc.808e, length 43
The only frames received are spanning tree frames, from the switch.
In comparison, this is what can be captured on a node in the production VLAN, in only a one second time:
14:27:43.920934 IP paravance-60.rennes.grid5000.fr.38784 > dns.rennes.grid5000.fr.domain: 65121+ PTR? 5.98.16.172.in-addr.arpa. (42)
14:27:43.921384 IP dns.rennes.grid5000.fr.domain > paravance-60.rennes.grid5000.fr.38784: 65121* 1/1/0 PTR parapide-5.rennes.grid5000.fr. (103)
14:27:43.921510 IP paravance-60.rennes.grid5000.fr.49250 > dns.rennes.grid5000.fr.domain: 48890+ PTR? 111.111.16.172.in-addr.arpa. (45)
14:27:43.921816 IP dns.rennes.grid5000.fr.domain > paravance-60.rennes.grid5000.fr.49250: 48890* 1/1/0 PTR kadeploy.rennes.grid5000.fr. (104)
14:27:44.017208 ARP, Request who-has parapide-5.rennes.grid5000.fr tell dns.rennes.grid5000.fr, length 46
14:27:44.201278 IP6 fe80::214:4fff:feca:9470 > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
14:27:44.201416 IP paravance-60.rennes.grid5000.fr.34416 > dns.rennes.grid5000.fr.domain: 7912+ PTR? 6.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.f.f.ip6.arpa. (90)
14:27:44.284641 ARP, Request who-has parapide-9.rennes.grid5000.fr tell kadeploy.rennes.grid5000.fr, length 46
14:27:44.307171 ARP, Request who-has parapide-5.rennes.grid5000.fr tell metroflux.rennes.grid5000.fr, length 46
14:27:44.398978 IP dns.rennes.grid5000.fr.domain > paravance-60.rennes.grid5000.fr.34416: 7912 NXDomain 0/1/0 (160)
Here we see ARP requests, DNS messages, multicast reports…
Communication without routing: OpenVSwitch
- Setup
Our goal is to allow 2 nodes from 2 different (physical) VLANs to communicate without routing. To do this, nodes from each side must be in the same IP network. For that, our two layer 2 VLANs must be bridged. In this tutorial, we will not use Linux bridge (the brctl
command), but OpenVSwitch, which provides enhanced functionalities.
First we need to change the IP of node_nancy to an IP in the local VLAN's subnet (but different from the IPs of node1 and node2 of course) (192.168.192.0/20 in our case, we can use 192.168.200.2 for instance). Then restart the networking service.
We already installed OpenVSwitch on our router node, node2, we can now use it.
Warning
node2 will lose its IP address, so the SSH session will be lost, use Kaconsole instead for the following
- Create the bridge
- Remove IPs on both interfaces eth0 and eth1:
Add eth0 and eth1 to the bridge:
- Tests
Now that it is done, you should be able to ping node_nancy (with its new IP!).
Also if you run traceroute, you will notice that there is only one hop between node1 and node_nancy.
- Flow control
You can use openVSwitch to manage flows, for example you can DROP all packets from an IP with this command:
"in_port=1" refers to eth0 in our case. You can list all the ports in the bridge with:
You can also DROP all pings to a specific IP from a specific port:
And the following command will display all your flow rules :
Note
If you want to know more about flow syntax go to this man page and look for the "Flow Syntax" paragraph
Automating the configuration with TopoMaker
TopoMaker is a totally new tool on Grid'5000 (software is still in beta testing for now), which allows to automate all steps of the "Topology setup" part (except the last paragraph about routing).
There is no documentations yet, but we will start from an example, to build the same topology as in the first paragraphs.
TopoMaker takes a Rspec XML file as input. That input is actually inspired by the syntax used in CloudLab (another experimental testbed). Here is the example we will use:
<?xml version="1.0" encoding="UTF-8"?>
<rspec>
<node client_id="node-2">
<interface client_id="interface-0"/> <!--No IP precised equals to DHCP-->
<interface client_id="interface-1">
<ip address="192.168.1.3" type="ipv4" netmask="255.255.255.0" />
</interface>
<sliver_type name="raw-pc">
<disk_image name="jessie-x64-min" /> <!--This is just for the example because default OS chosen by topomaker is jessie-x64-min -->
</sliver_type>
<auto_install>
<apt name="tcpdump"/>
<apt name="openvswitch-switch"/>
</auto_install>
</node>
<node client_id="node-1"> <!--Here we will have DHCP on jessie x64 min-->
<interface client_id="interface-2"/>
</node>
<node client_id="node-3"> <!--node 3 is equivalent to our node_nancy here it will be on the same site, as we saw it doesn't matter with global vlan-->
<interface client_id="interface-4">
<ip address="192.168.1.1" type="ipv4" netmask="255.255.255.0" />
</interface>
</node>
<link client_id="link-0"> <!--link are equivalent to VLAN-->
<interface_ref client_id="interface-0" />
<interface_ref client_id="interface-2" />
</link>
<link client_id="link-1">
<interface_ref client_id="interface-1" />
<interface_ref client_id="interface-4" />
</link>
</rspec>
As you can see, each part of this XML file is totally independent from the nodes you reserved. You can put the name of your choice for nodes and interfaces names. It just have to be relevant between names for interfaces and names for links.
TopoMaker isn't provided as a Grid'5000 tool yet, so we will download the source code and use it directly.
TopoMaker is developed in Ruby and requires some gems, which are not installed in the environement. We install them:
TopoMaker does not do the reservation for you. You should have a running job and provide its jobid.
The TopoMaker command line is as follows:
You can add the "-v" option to show more information about the progress of the execution (the script might look like stuck sometime). In the end, a yaml file is prompted with all the required information, like IP or hostname.
TopoMaker doesn't know about DNS in kavlan-local (and IP are not retrieved), so the rules is the same as in the tutorial if you used DHCP: "hostname-kavlan-ID.rennes.grid5000.fr" from the gateway to connect to the node.
Once the script is done, you can do the same tests as above, and do the modification yourself to use openVSwitch with TopoMaker.