Network reconfiguration tutorial: Difference between revisions
(→Nancy) |
|||
Line 95: | Line 95: | ||
{{Term|location=nancy:frontend|cmd=<code class="command">ssh</code> root@<code class="replace">Hostname_node_nancy</code> "apt-get --yes install at && echo 'service networking restart' | at now + 1 minute" && kavlan -s -i <code class="replace">Global_Vlan_Id</code> -m <code class="replace">Hostname_Node_Nancy</code>}} | {{Term|location=nancy:frontend|cmd=<code class="command">ssh</code> root@<code class="replace">Hostname_node_nancy</code> "apt-get --yes install at && echo 'service networking restart' | at now + 1 minute" && kavlan -s -i <code class="replace">Global_Vlan_Id</code> -m <code class="replace">Hostname_Node_Nancy</code>}} | ||
<pre style="color: red">(bpichot) make more clear which node is node_Nancy?</pre> | |||
===Rennes=== | ===Rennes=== |
Revision as of 19:12, 12 January 2016
Note | |
---|---|
This page is actively maintained by the Grid'5000 team. If you encounter problems, please report them (see the Support page). Additionally, as it is a wiki page, you are free to make minor corrections yourself if needed. If you would like to suggest a more fundamental change, please contact the Grid'5000 team. |
Introduction
This TP aims to discover a method to configure a network in Grid'5000 using KaVLAN.
KaVLAN is a tool on Grid'5000 which allows to the user to manage VLANs in the platform. It edits switch configuration to change the VLAN number of the port corresponding to the interface of a node. This method permits complete layer 2 isolation.
Three kinds of VLANs are available on Grid'5000, you can find more information on the page KaVLAN. In this TP, we will use only global Vlan and local Vlan.
Initially, we will set up a simple topology with 2 VLANs, a global and a local. In each VLAN there will be at least one node, and one interface of a node with 2 interfaces.
Set up topology
Reservations
A global VLAN is all over Grid'5000, so we have to reserve it on only one site. We will use Rennes and Nancy sites.
rennes:frontend :
|
oarsub -l {"type='kavlan-global'"}/vlan=1+{"type='kavlan-local'"}/vlan=1+{"cluster='paravance'"}/nodes=3,walltime=3 -I -t deploy |
With this reservation, we have 1 kavlan-local, 1 kavlan-global and 3 nodes on paravance cluster. Paravance is a cluster of nodes with 2 interfaces.
Get your Vlans ID :
How do we know which VLAN is global and which one is local ? It's simple, it's written in KaVLAN (look at the first diagram ;)) :
- kavlan-local : [1-3]
- kavlan : [4-9]
- global : [10-20]
Deployment
Now we will deploy our nodes with debian jessie minimal :
During the deployment you can reserve and deploy a node on Nancy in another terminal :
We will install 'at' and 'tcpdump' on each node using TakTuk :
rennes:node :
|
taktuk -s -l root -f $OAR_FILE_NODES broadcast exec [ "apt-get update; apt-get --yes install at tcpdump" ]
|
Since jessie, the default sshd configuration doesn't allow password authentication, then you can't connect to a deployed node from an other node. There are two solutions:
kaconsole
Kaconsole is a tool provided in Grid'5000, basically it allows to connect to a node the same way you would connect to it with a screen and a keyboard.
So, you can connect on the node with the credentials "root":"grid5000".
Add SSH key pair on each node
You can just use this small script from the reservation prompt :
#! /bin/bash temp=$(mktemp -d XXXXX) ssh-keygen -t rsa -f $temp/id_rsa -P "" for i in $(uniq $OAR_NODEFILE) ; do scp $temp/id_rsa root@$i:.ssh/id_rsa && ssh-copy-id -i $temp/id_rsa.pub root@$i done rm -r $temp
It will generate a temporary RSA key pair, and copy the private one as well as allow the public one on each node.
Network configuration
Nancy
We will put a static IP on our node on Nancy and put it in the VLAN. So configure the first interface (we will assume this is eth0) with the following in /etc/network/interfaces:
auto eth0 iface eth0 inet static address 192.168.1.1 netmask 255.255.255.0
nancy:frontend :
|
ssh root@Hostname_node_nancy "apt-get --yes install at && echo 'service networking restart' | at now + 1 minute" && kavlan -s -i Global_Vlan_Id -m Hostname_Node_Nancy |
(bpichot) make more clear which node is node_Nancy?
Rennes
Let's call our 3 nodes on Rennes node1, node2 and node3. Of course during the TP node1=<hostname of your first node>. For example node1=paravance-23.
We will put node1 in the local VLAN and restart the networking service to get a new IP address (there is a DHCP server in kavlan-local).
rennes:frontend :
|
ssh root@node1 "echo 'service networking restart' | at now + 1 minute" && kavlan -s -i Local_Vlan_Id -m node1 |
Warning | |
---|---|
Later we will configure an OpenVSwitch on node2, so install it with 'apt-get install openvswitch-switch' before putting node2 in another VLAN than (production) DEFAULT one |
Now we will put a node between the local and global vlan using node2. We have to set up the second interface. We know eth1 is linked with the API : https://api.grid5000.fr/sid/sites/rennes/clusters/paravance/nodes/paravance-1.json?pretty
So we have to add to /etc/network/interfaces on node2 :
auto eth1 iface eth1 inet static address 192.168.1.2 netmask 255.255.255.0
We need to put the first interface in the local-vlan and the second in the global :
rennes:frontend :
|
ssh root@node2 "echo 'service networking restart' | at now + 1 minute" && kavlan -s -i Local_Vlan_Id -m node1 && kavlan -s -i Global_Vlan_Id -m node2-eth1 |
Now we have :
[prod <-]--kavlan-ID--[-> local-vlan] [local-vlan <--node1-->] [local-vlan <-]--node2--[-> global-vlan] [global-vlan <--node_nancy-->]
Now you should be able to ssh connect node2 from kavlan gateway, and ping 192.168.1.1 (node_nancy).
Routing
node1 and node_nancy are in two different VLAN, so currently it's impossible to send a packet between node1 and node_nancy. To do this, we will set route between the network in the local vlan and the network in the global VLAN (192.168.1.0/24)
Note | |
---|---|
To know the network address in the local vlan, connect to node1, node2 or kavlan-ID and enter the command : ip route |
Packets can't change of network without router between the vlans, so we will use our gateway (node2) to do this. Firstly we have to allow ip forwarding on node2.
Now, if a IP packet with known network destination is sent to node2, it will be forwarded to the destination network.
But node1 don't know how to reach node_nancy network and vice versa. We will add a route to each node.
Topology test
We will check if packet pass through our gateway :
Open two terminal :
On the first terminal you should see the ICMP packet are forwarded on the node2 :
IP 192.168.200.7 > 192.168.1.1: ICMP echo request, id 4270, seq 1, length 64 IP 192.168.1.1 > 192.168.200.7: ICMP echo reply, id 4270, seq 1, length 64 IP 192.168.200.7 > 192.168.1.1: ICMP echo request, id 4270, seq 2, length 64 IP 192.168.1.1 > 192.168.200.7: ICMP echo reply, id 4270, seq 2, length 64 IP 192.168.200.7 > 192.168.1.1: ICMP echo request, id 4270, seq 3, length 64 IP 192.168.1.1 > 192.168.200.7: ICMP echo reply, id 4270, seq 3, length 64
If you disable ip forwarding on node2, nothing will happen in tcpdump and ping will fail !
We can check the route with traceroute from node1 (or from node_nancy whatever) :
And the result is :
traceroute to 192.168.1.1 (192.168.1.1), 30 hops max, 60 byte packets 1 192.168.200.8 (192.168.200.8) 0.136 ms 0.122 ms 0.115 ms 2 192.168.1.1 (192.168.1.1) 25.612 ms 25.617 ms 25.611 ms
The first jump is from node1 to node2, and the second node is from node2 to node_nancy.
With tcpdump we can check the complete isolation of nodes from the production Vlan (or any other) :
On my node1 (in 12 seconds):
14:26:15.107927 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 82bd.8c:60:4f:47:6c:bc.808e, length 43 14:26:17.109436 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 82bd.8c:60:4f:47:6c:bc.808e, length 43 14:26:19.108669 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 82bd.8c:60:4f:47:6c:bc.808e, length 43 14:26:21.108675 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 82bd.8c:60:4f:47:6c:bc.808e, length 43 14:26:23.108669 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 82bd.8c:60:4f:47:6c:bc.808e, length 43 14:26:25.108654 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 82bd.8c:60:4f:47:6c:bc.808e, length 43 14:26:27.108674 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 82bd.8c:60:4f:47:6c:bc.808e, length 43
We just have spanning tree frame from the switch.
On a node in production vlan (in only one second) :
14:27:43.920934 IP paravance-60.rennes.grid5000.fr.38784 > dns.rennes.grid5000.fr.domain: 65121+ PTR? 5.98.16.172.in-addr.arpa. (42) 14:27:43.921384 IP dns.rennes.grid5000.fr.domain > paravance-60.rennes.grid5000.fr.38784: 65121* 1/1/0 PTR parapide-5.rennes.grid5000.fr. (103) 14:27:43.921510 IP paravance-60.rennes.grid5000.fr.49250 > dns.rennes.grid5000.fr.domain: 48890+ PTR? 111.111.16.172.in-addr.arpa. (45) 14:27:43.921816 IP dns.rennes.grid5000.fr.domain > paravance-60.rennes.grid5000.fr.49250: 48890* 1/1/0 PTR kadeploy.rennes.grid5000.fr. (104) 14:27:44.017208 ARP, Request who-has parapide-5.rennes.grid5000.fr tell dns.rennes.grid5000.fr, length 46 14:27:44.201278 IP6 fe80::214:4fff:feca:9470 > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28 14:27:44.201416 IP paravance-60.rennes.grid5000.fr.34416 > dns.rennes.grid5000.fr.domain: 7912+ PTR? 6.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.f.f.ip6.arpa. (90) 14:27:44.284641 ARP, Request who-has parapide-9.rennes.grid5000.fr tell kadeploy.rennes.grid5000.fr, length 46 14:27:44.307171 ARP, Request who-has parapide-5.rennes.grid5000.fr tell metroflux.rennes.grid5000.fr, length 46 14:27:44.398978 IP dns.rennes.grid5000.fr.domain > paravance-60.rennes.grid5000.fr.34416: 7912 NXDomain 0/1/0 (160)
We have ARP request, DNS message, multicast report, ...
Communication without routing : OpenVSwitch
Previously we already installed openswitch on our "gateway" node (node2). We will use it !
Our objective is to allow 2 nodes from 2 different VLANs to communicate without routing. To do this, nodes from each side must be in the same network, so change the IP configuration of node_nancy by something in agreement with the subnet of the vlan local but different of node1 and node 2(192.168.192.0/20 in our situation => 192.168.200.2 for example) and restart networking service.
Now we have to set up the OpenVSwitch config on node2.
Warning | |
---|---|
node2 will lose its IP address, so the SSH session will be lose, use Kaconsole instead for this part |
- Create the bridge
- Delete IP from eth0 and eth1
Add interfaces eth0 and eth1 to the bridge :
It's done, you should be able to ping node_nancy (with its new IP !). If you do a traceroute, you will notice that : There is only one jump : node1 => node_nancy.
You can use openVSwitch to manage flows, for example you can DROP all packets from an IP with this command :
"in_port=1" equal to eth0 in our case. You can list all port with :
Limits are really uncommon, for example you can DROP all ping to an IP from a specific port :
You can also display all your flow rules :
Note | |
---|---|
If you want to know more about flow syntax go to this page and search "Flox Syntax" paragraph |
Topo_maker
Topo_maker is a totally new tool on Grid'5000 (consider this as beta testing for the moment) it allows you to do all step of the part "set up topology" (except routing). There is no official documentations yet, but we will start from an example to do the same topology as above.
topo_maker wait a Rspec XML file as input, it is inspired by the syntax used in CloudLab (another Cloud platform), this is the example we will use :
<?xml version="1.0" encoding="UTF-8"?> <rspec> <node client_id="node-2"> <interface client_id="interface-0"/> <!--No IP precised equals to DHCP--> <interface client_id="interface-1"> <ip address="192.168.1.3" type="ipv4" netmask="255.255.255.0" /> </interface> <sliver_type name="raw-pc"> <disk_image name="jessie-x64-min" /> <!--This is just for the example because default OS chosen by topomaker is jessie-x64-min --> </sliver_type> <auto_install> <apt name="tcpdump"/> <apt name="openvswitch-switch"/> </auto_install> </node> <node client_id="node-1"> <!--Here we will have DHCP on jessie x64 min--> <interface client_id="interface-2"/> </node> <node client_id="node-3"> <!--node 3 is equivalent to our node_nancy here it will be on the same site, as we saw it doesn't matter with global vlan--> <interface client_id="interface-4"> <ip address="192.168.1.1" type="ipv4" netmask="255.255.255.0" /> </interface> </node> <link client_id="link-0"> <!--link are equivalent to VLAN--> <interface_ref client_id="interface-0" /> <interface_ref client_id="interface-2" /> </link> <link client_id="link-1"> <interface_ref client_id="interface-1" /> <interface_ref client_id="interface-4" /> </link> </rspec>
As you can see each part of this XML file is totally independant of the nodes you reserved, you can put the name of your choice for nodes name and interface name. It just have to be coherent between names in interfaces and names in links.
TopoMaker isn't released yet, so we will download the source code and use it directly.
topo_maker is in ruby and some gem aren't installed (yet).
Topo_maker doesn't sub your reservation you should have a reservation and its job id. In the reservation prompt you can exec "echo $OAR_JOBID" else you can find your reservation in "oarstat -u".
Now, we will start topo_maker :
You could add "-v" option to show more information about the progress and avoid to think the script is stuck. At the end a yaml file is prompt with all informations needed like IP or hostname. Topo_maker doesn't know about DNS in kavlan-local (and IP aren't get back) so the rules is the same as in the TP if you used DHCP : "hostname-kavlan-ID.rennes.grid5000.fr" from the gateway to connect to the node.
One the script is done, you can do the same test as above, and do yourself the modification to use openVSwitch with TopoMaker to exercise yourself.