KaVLAN: Difference between revisions
m (→1: Local VLAN) |
|||
| (15 intermediate revisions by 5 users not shown) | |||
| Line 1: | Line 1: | ||
{{Maintainer|Nicolas Niclausse}} | {{Maintainer|Nicolas Niclausse}} | ||
{{Maintainer|Pierre Neyron}} | |||
{{Portal|User}} | {{Portal|User}} | ||
{{Portal|Tutorial}} | {{Portal|Tutorial}} | ||
| Line 5: | Line 6: | ||
{{Status|In production}} | {{Status|In production}} | ||
{{Pages|KaVLAN}} | {{Pages|KaVLAN}} | ||
{{TutorialHeader}} | |||
__FORCETOC__ | __FORCETOC__ | ||
| Line 22: | Line 24: | ||
''It is however important to note that KaVLAN does not guarantee performance isolation: on sites with a hierarchical network (such as [[Nancy:Network|Nancy]]), inter-switch links may indeed be shared between various VLANs/experiments.'' | ''It is however important to note that KaVLAN does not guarantee performance isolation: on sites with a hierarchical network (such as [[Nancy:Network|Nancy]]), inter-switch links may indeed be shared between various VLANs/experiments.'' | ||
For experimentations involving network reconfiguration, [[KaVLAN]] is to be used together with | For experimentations involving network reconfiguration, [[KaVLAN]] is to be used together with OAR and Kadeploy (for the resources reservation and to gain control over the operating system and network configuration of the nodes) | ||
= The 3 KaVLAN VLAN types = | = The 3 KaVLAN VLAN types = | ||
| Line 36: | Line 34: | ||
== 1: Local VLAN == | == 1: Local VLAN == | ||
From the IP routing point of view, | From the IP routing point of view, a ''local VLAN'' is completely '''isolated''' from the rest of Grid'5000. '''No IP routing is configured in any router of the infrastructure'''. Therefore, to reach your nodes inside that kind of VLAN, the Grid'5000 infrastructure provides a special host you can hop by: '''the SSH gateway of the VLAN'''. For each local VLAN, the hostname of that SSH gateway is: ''kavlan-<code class="replace">ID</code>''. | ||
| Line 42: | Line 40: | ||
The figure below shows two jobs using KaVLAN: each job has its nodes isolated in a ''local VLAN'' ('''<font color="green">green</font>''' and '''<font color="purple">purple</font>'''). The other nodes are all in the default VLAN ('''<font color="red">red</font>'''). The only way to reach the isolated nodes is to hop by the VLAN's ''SSH gateway'' machine (kavlan-1 and kavlan-2 in the figure). Technically speaking, the ''SSH gateway'' has two Ethernet interfaces: one in the default VLAN and one in the dedicated VLAN. | The figure below shows two jobs using KaVLAN: each job has its nodes isolated in a ''local VLAN'' ('''<font color="green">green</font>''' and '''<font color="purple">purple</font>'''). The other nodes are all in the default VLAN ('''<font color="red">red</font>'''). The only way to reach the isolated nodes is to hop by the VLAN's ''SSH gateway'' machine (kavlan-1 and kavlan-2 in the figure). Technically speaking, the ''SSH gateway'' has two Ethernet interfaces: one in the default VLAN and one in the dedicated VLAN. Another way to reach an isolated node is to use the <code class='command'>kaconsole</code> command. | ||
[[Image:kavlan.png|450px|center|thumbnail|KaVLAN architecture: 2 jobs running KaVLAN]] | [[Image:kavlan.png|450px|center|thumbnail|KaVLAN architecture: 2 jobs running KaVLAN]] | ||
{{Note|text=Please | {{Note|text=Please note that: | ||
* as your nodes are isolated from the rest of Grid'5000, NFS mounts of /home partition is not possible. Therefore, '''Grid'5000 environments that mount /home partition (-nfs, -big, - | * as your nodes are isolated from the rest of Grid'5000, NFS mounts of /home partition is not possible. Therefore, '''Grid'5000 environments that mount /home partition (-nfs, -big, -std) may fail to boot'''}} | ||
== 2: Routed VLAN == | == 2: Routed VLAN == | ||
| Line 68: | Line 66: | ||
{{Note|text=Please mind that there is not performance isolation between all ''global VLANs'' and also Grid'5000 inter-site VLAN (backbone VLAN). All share the same inter-site ''physical'' link}} | {{Note|text=Please mind that there is not performance isolation between all ''global VLANs'' and also Grid'5000 inter-site VLAN (backbone VLAN). All share the same inter-site ''physical'' link}} | ||
{{Note|text=Inside Grid'5000's vlan API, the vlan type kavlan-global-remote can also be found, this is the same type as kavlan-global. This difference is due to kavlan's internal machinery.|inline=1}} | |||
= Reserving a VLAN = | = Reserving a VLAN = | ||
| Line 82: | Line 82: | ||
{{Term|location=frontend|cmd=<code class="command">kavlan</code> -V -j <code class="replace">JOBID</code>}} | {{Term|location=frontend|cmd=<code class="command">kavlan</code> -V -j <code class="replace">JOBID</code>}} | ||
Either ways, you should get | Either ways, you should get a VLAN ID integer in the ''<1-3>'' range for ''local VLANs'', ''<4-9>'' for ''routed VLANs'', and greater than 10 for ''global VLANs'' (only one global VLAN ID is available per site, that should be the one of the site you are connected to). | ||
| Line 90: | Line 90: | ||
= Setting up the VLAN = | = Setting up the VLAN = | ||
Configuring the VLANs is done with the <code class="command">kavlan</code> command. | Configuring the VLANs is done with the '''<code class="command">kavlan</code>''' command. | ||
All the options of the command can be show using ''--help'', as follows: | All the options of the command can be show using ''--help'', as follows: | ||
| Line 118: | Line 118: | ||
</pre> | </pre> | ||
So, once you have a ''kavlan'' job running, and know your vlan ID, you can use the <code class="command">kavlan</code> to put some network interfaces of your nodes in your VLAN (and later, back into the default VLAN) at anytime during the lifetime of your job. | So, once you have a ''kavlan'' job running, and know your vlan ID, you can use the '''<code class="command">kavlan</code>''' command to put some network interfaces of your nodes in your VLAN (and later, back into the default VLAN) at anytime during the lifetime of your job. | ||
In case of a node with multiple cabled network interfaces, each of them can be used, with the following naming: | In case of a node with multiple cabled network interfaces, each of them can be used, with the following naming: | ||
* for the default interface: <code class="replace">hostname-X</code>-kavlan-<code class="replace">ID</code> | * for the default interface: <code class="replace">hostname-X</code>-kavlan-<code class="replace">ID</code> | ||
* for other interfaces: <code class="replace">hostname-X</code>-eth<code class="replace">Y</code>-kavlan-<code class="replace">ID</code> | * for other interfaces: <code class="replace">hostname-X</code>-eth<code class="replace">Y</code>-kavlan-<code class="replace">ID</code> | ||
{{Note|text=You may notice that the hostname for secondaries interfaces is formed like this "<code class="replace">hostname-X</code>-eth<code class="replace">Y</code>-kavlan-<code class="replace">ID</code>" while the name of the interface is the system is "<code class="replace">enoY</code> or <code class="replace">enpYsZ</code>". | |||
It's due to changes in the naming of interfaces since debian9 (see https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/).<br/> | |||
Kavlan hostname still uses old interface names, if you're not sure which name correspond to which interface, both naming (old and new) are describes in the api. | |||
For exemple, '''grisou-2-eth1-kavlan-1.nancy.grid5000.fr''' will correspond to interface '''eno2''' on '''grisou-2''' ( https://api.grid5000.fr/stable/sites/nancy/clusters/grisou/nodes/grisou-2.json ) | |||
}} | |||
A DHCP service is provided in all | ;A DHCP service is provided in all VLANs (local, routed and global): | ||
Once you have put network interfaces of some nodes in a VLAN, you can down-up them, or restart the networking service of the operating system (with kaconsole3 for instance, or using a command line with some bash magic like '| at now + 1 minute', to run the command asynchronously and overcome the network disconnection that will occur), or reboot the node (with kareboot3) in order to get a relevant IP for the VLAN. | |||
If needed for your experiment, please note that the <code class="command">kavlan</code> command allows to deactivate the DHCP service in a VLAN. | If needed for your experiment, please note that the '''<code class="command">kavlan</code>''' command allows to deactivate the DHCP service in a VLAN. | ||
Reminder: for local VLANs, you are also allowed to ssh to the VLAN's SSH gateway, which is named kavlan-<code class='replace'>ID</code>. | |||
Please look at the KaVLAN | Please look at the other KaVLAN pages for examples of usage (look at the '''see-also dialog box''' at the top of the page). | ||
Latest revision as of 11:14, 21 October 2022
|
Note |
|---|---|
This page is actively maintained by the Grid'5000 team. If you encounter problems, please report them (see the Support page). Additionally, as it is a wiki page, you are free to make minor corrections yourself if needed. If you would like to suggest a more fundamental change, please contact the Grid'5000 team. | |
Overview
KaVLAN provides network isolation capabilities for Grid'5000 users' experimentations, via a high-level, user-driven interface to VLANs (802.1Q).
Said differently: KaVLAN allows users to manage VLANs for the network connection of their Grid'5000 nodes.
Behind the scenes, KaVLAN actually changes the configuration of the network switches of Grid'5000 infrastructure, to set the VLAN membership (VLAN ID) for the ports which are cabled to the network interfaces of one or more nodes.
The benefit is a complete level 2 isolation for users' experiments.
It is however important to note that KaVLAN does not guarantee performance isolation: on sites with a hierarchical network (such as Nancy), inter-switch links may indeed be shared between various VLANs/experiments.
For experimentations involving network reconfiguration, KaVLAN is to be used together with OAR and Kadeploy (for the resources reservation and to gain control over the operating system and network configuration of the nodes)
The 3 KaVLAN VLAN types
3 types of VLANs are available for users in Grid'5000: local, routed and global.
| KaVLAN name in OAR | type | first id | last id |
|---|---|---|---|
| kavlan-local | local | 1 | 3 |
| kavlan | routed | 4 | 9 |
| kavlan-global | global | 10 | 21 |
See the 2 schemas on the right of this page, which illustrate KaVLAN big picture and architecture.
1: Local VLAN
From the IP routing point of view, a local VLAN is completely isolated from the rest of Grid'5000. No IP routing is configured in any router of the infrastructure. Therefore, to reach your nodes inside that kind of VLAN, the Grid'5000 infrastructure provides a special host you can hop by: the SSH gateway of the VLAN. For each local VLAN, the hostname of that SSH gateway is: kavlan-ID.
Then you can connect to any of your nodes within the VLAN using hostnames such as hostname-X-kavlan-ID (adding the suffix -kavlan- + the VLAN_ID to the regular hostname), for instance from the SSH gateway of the VLAN, or from node to node (with the default provided DNS configuration in the VLAN).
The figure below shows two jobs using KaVLAN: each job has its nodes isolated in a local VLAN (green and purple). The other nodes are all in the default VLAN (red). The only way to reach the isolated nodes is to hop by the VLAN's SSH gateway machine (kavlan-1 and kavlan-2 in the figure). Technically speaking, the SSH gateway has two Ethernet interfaces: one in the default VLAN and one in the dedicated VLAN. Another way to reach an isolated node is to use the kaconsole command.
|
|
Note |
|---|---|
Please note that:
| |
2: Routed VLAN
Unlike local VLANs which are isolated, routed VLANs are not isolated at the layer 3: IP packets are routed. Therefore you can reach the nodes inside a routed VLAN from the rest of Grid5000 (e.g. from the default VLAN, or from another routed VLAN). No need here for a hop by a SSH gateway, as it is the case for local VLANs.
Nodes in the VLAN are reachable with the following hostname: hostname-X-kavlan-ID (same naming scheme as for local VLANs), from the frontends of the sites for instance.
3: Global VLAN
Global VLANs are VLANs which spread on all grid5000 sites. Therefore you can configure nodes of different sites in the same global VLAN, i.e. in a same Ethernet network (no inter-site IP routing required, nodes in a global VLAN use a same broadcast domain).
(underneath they use the IEEE 802.1ad encapsulation, also known as QinQ to provide a same layer 2 network for all sites.)
There is exactly 1 and only 1 global VLAN provided by site. If that VLAN is already reserved by another user, you can try to get one from another site. Reservation must be made on the site of the global VLAN.
Since it is a same layer 2 network, no routing between the nodes which are placed in a global VLAN is required (even from site to site).
To reach nodes inside a global VLAN from outside, routing is configured on the router of the site where the global VLAN is reserved.
The hostnames of nodes within a VLAN follow the same scheme as above: hostname-X-kavlan-ID.
|
|
Note |
|---|---|
Please mind that there is not performance isolation between all global VLANs and also Grid'5000 inter-site VLAN (backbone VLAN). All share the same inter-site physical link | |
|
|
Note |
|---|---|
Inside Grid'5000's vlan API, the vlan type kavlan-global-remote can also be found, this is the same type as kavlan-global. This difference is due to kavlan's internal machinery. | |
Reserving a VLAN
Using KaVLAN requires to works with deploy reservations because it necessarily involves reconfiguring the network stack of the operating system of the nodes.
To obtain both nodes and a VLAN, you must reserve kavlan resources (VLAN-IDs) with OAR using the oarsub command. As shown in the table above, there are 3 kinds of resources defined in OAR for VLANs: kavlan, kavlan-local, kavlan-global. For example, if you need 3 nodes and a local VLAN, you can run:
|
frontend:
|
oarsub -t deploy -l {"type='kavlan-local'"}/vlan=1+/nodes=3 -I |
Then you can get the ID of your VLAN using the kavlan command
|
|
frontend:
|
kavlan -V |
If you need to run that command from outside the shell which is started by OAR for your reservation, you have to give the OAR JOBID.
|
|
frontend:
|
kavlan -V -j JOBID |
Either ways, you should get a VLAN ID integer in the <1-3> range for local VLANs, <4-9> for routed VLANs, and greater than 10 for global VLANs (only one global VLAN ID is available per site, that should be the one of the site you are connected to).
See below the KaVLAN ID, and associated IP subnets (served by DHCP in the VLANs)
- Local VLANs (non-routed)
| Site | KAVLAN-1 | KAVLAN-2 | KAVLAN-3 |
|---|---|---|---|
| All | 192.168.192.0/20 |
192.168.208.0/20 |
192.168.224.0/20
|
- Routed VLANs
| Site | KAVLAN-4 | KAVLAN-5 | KAVLAN-6 | KAVLAN-7 | KAVLAN-8 | KAVLAN-9 |
|---|---|---|---|---|---|---|
10.0.0.0/18 |
10.0.64.0/18 |
10.0.128.0/18 |
10.0.192.0/18 |
10.1.0.0/18 |
10.1.64.0/18
| |
| Grenoble | 10.4.0.0/18 |
10.4.64.0/18 |
10.4.128.0/18 |
10.4.192.0/18 |
10.5.0.0/18 |
10.5.64.0/18
|
| Lille | 10.8.0.0/18 |
10.8.64.0/18 |
10.8.128.0/18 |
10.8.192.0/18 |
10.9.0.0/18 |
10.9.64.0/18
|
| Lyon | 10.12.0.0/18 |
10.12.64.0/18 |
10.12.128.0/18 |
10.12.192.0/18 |
10.13.0.0/18 |
10.13.64.0/18
|
| Nancy | 10.16.0.0/18 |
10.16.64.0/18 |
10.16.128.0/18 |
10.16.192.0/18 |
10.17.0.0/18 |
10.17.64.0/18
|
10.20.0.0/18 |
10.20.64.0/18 |
10.20.128.0/18 |
10.20.192.0/18 |
10.21.0.0/18 |
10.21.64.0/18
| |
| Rennes | 10.24.0.0/18 |
10.24.64.0/18 |
10.24.128.0/18 |
10.24.192.0/18 |
10.25.0.0/18 |
10.25.64.0/18
|
| Toulouse | 10.28.0.0/18 |
10.28.64.0/18 |
10.28.128.0/18 |
10.28.192.0/18 |
10.29.0.0/18 |
10.29.64.0/18
|
| Sophia | 10.32.0.0/18 |
10.32.64.0/18 |
10.32.128.0/18 |
10.32.192.0/18 |
10.33.0.0/18 |
10.33.64.0/18
|
| Strasbourg |
10.36.0.0/18 |
10.36.64.0/18 |
10.36.128.0/18 |
10.36.192.0/18 |
10.37.0.0/18 |
10.37.64.0/18
|
| Luxembourg | 10.40.0.0/18 |
10.40.64.0/18 |
10.40.128.0/18 |
10.40.192.0/18 |
10.41.0.0/18 |
10.41.64.0/18
|
| Nantes | 10.44.0.0/18 |
10.44.64.0/18 |
10.44.128.0/18 |
10.44.192.0/18 |
10.45.0.0/18 |
10.45.64.0/18
|
| Louvain | 10.48.0.0/18 |
10.48.64.0/18 |
10.48.128.0/18 |
10.48.192.0/18 |
10.49.0.0/18 |
10.49.64.0/18
|
|
|
Note |
|---|---|
At the end of each network, address x.x.x.253 is used by Kavlan server | |
- Global VLANs
| Site | Global Vlan | Subnet | Router IP |
|---|---|---|---|
KAVLAN-10 |
10.3.192.0/18 |
10.3.255.254
| |
| Grenoble | KAVLAN-11 |
10.7.192.0/18 |
10.7.255.254
|
| Lille | KAVLAN-12 |
10.11.192.0/18 |
10.11.255.254
|
| Lyon | KAVLAN-13 |
10.15.192.0/18 |
10.15.255.254
|
| Nancy | KAVLAN-14 |
10.19.192.0/18 |
10.19.255.254
|
KAVLAN-15 |
10.23.192.0/18 |
10.23.255.254
| |
| Rennes | KAVLAN-16 |
10.27.192.0/18 |
10.27.255.254
|
| Toulouse | KAVLAN-17 |
10.31.192.0/18 |
10.31.255.254
|
| Sophia | KAVLAN-18 |
10.35.192.0/18 |
10.35.255.254
|
| Strasbourg |
KAVLAN-19 |
10.39.192.0/18 |
10.39.255.254
|
| Luxembourg | KAVLAN-20 |
10.43.192.0/18 |
10.43.255.254
|
| Nantes | KAVLAN-21 |
10.47.192.0/18 |
10.47.255.254
|
| Louvain | KAVLAN-22 |
10.51.192.0/18 |
10.51.255.254
|
- IP subnet assignments for the sites within a global VLANs
A global VLAN is a /18 subnet (16382 IP addresses). It is split so that every site gets one /23 (510 ip) in the global VLAN address space.
Example for the global VLAN of Lille, KAVLAN-12, whose address space is 10.11.192.0/18:
Bordeaux:10.11.192.1→10.11.193.254- Grenoble:
10.11.194.1→10.11.195.254 - Lille:
10.11.196.1→10.11.197.254 - Lyon:
10.11.198.1→10.11.199.254 - Nancy:
10.11.200.1→10.11.201.254 Orsay:10.11.202.1→10.11.203.254- Rennes:
10.11.204.1→10.11.205.254 - Toulouse:
10.11.206.1→10.11.207.254 - Sophia:
10.11.208.1→10.11.209.254 - Strasbourg
Reims:10.11.210.1→10.11.211.254 - Luxembourg:
10.11.212.1→10.11.213.254 - Nantes:
10.11.214.1→10.11.215.254
(More info in the Network page)
Setting up the VLAN
Configuring the VLANs is done with the kavlan command.
All the options of the command can be show using --help, as follows:
# kavlan --help
Usage: kavlan [options]
Specific options:
-i, --vlan-id N set VLAN ID (integer or DEFAULT)
-C, --ca-cert CA CA certificate
-c, --client-cert CERT client certificate
-k, --client-key KEY client key
-l, --get-nodelist Show nodenames in the given vlan
-e, --enable-dhcp Start DHCP server
-d, --disable-dhcp Stop DHCP server
-V, --show-vlan-id Show vlan id of job (needs -j JOBID)
-g, --get-vlan Show vlan of nodes
-s, --set-vlan Set vlan of nodes
-j, --oar-jobid JOBID OAR job id
-m, --machine NODE set nodename (several -m are OK)
-f, --filename NODEFILE read nodes from a file
-u, --user USERNAME username
-v, --[no-]verbose Run verbosely
-q, --[no-]quiet Run quietly
--[no-]debug Run with debug output
-h, --help Show this message
--version Show version
So, once you have a kavlan job running, and know your vlan ID, you can use the kavlan command to put some network interfaces of your nodes in your VLAN (and later, back into the default VLAN) at anytime during the lifetime of your job.
In case of a node with multiple cabled network interfaces, each of them can be used, with the following naming:
- for the default interface:
hostname-X-kavlan-ID - for other interfaces:
hostname-X-ethY-kavlan-ID
|
|
Note |
|---|---|
You may notice that the hostname for secondaries interfaces is formed like this " It's due to changes in the naming of interfaces since debian9 (see https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/). | |
- A DHCP service is provided in all VLANs (local, routed and global)
Once you have put network interfaces of some nodes in a VLAN, you can down-up them, or restart the networking service of the operating system (with kaconsole3 for instance, or using a command line with some bash magic like '| at now + 1 minute', to run the command asynchronously and overcome the network disconnection that will occur), or reboot the node (with kareboot3) in order to get a relevant IP for the VLAN.
If needed for your experiment, please note that the kavlan command allows to deactivate the DHCP service in a VLAN.
Reminder: for local VLANs, you are also allowed to ssh to the VLAN's SSH gateway, which is named kavlan-ID.
Please look at the other KaVLAN pages for examples of usage (look at the see-also dialog box at the top of the page).