IPv6

From Grid5000
Revision as of 18:50, 10 July 2024 by Bjonglez (talk | contribs) (→‎Current status: Update, we have reconfigurable firewall now)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search
See also: Hardware | KaVLAN | Network reconfiguration tutorial | Advanced KaVLAN | Advanced IPv6 | Reconfigurable firewall | Network emulation
Note.png Note

This page is actively maintained by the Grid'5000 team. If you encounter problems, please report them (see the Support page). Additionally, as it is a wiki page, you are free to make minor corrections yourself if needed. If you would like to suggest a more fundamental change, please contact the Grid'5000 team.

Basic IPv6 usage in Grid'5000

Grid'5000 supports IPv6 communication within the platform and with the rest of the Internet.

First step: obtaining an IPv6 address

By default, nodes don't have an IPv6 address. You need to request one using DHCPv6.

This can be done in the following way from the standard environment:

Terminal.png node:
sudo-g5k dhclient -6 br0

You should obtain a publicly routable IPv6 address in the prefix 2001:660:4406::/48. You can see it with:

Terminal.png node:
ip address show dev br0

which should return something similar to: 

8: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
   link/ether 98:be:94:68:72:34 brd ff:ff:ff:ff:ff:ff
   inet 172.16.24.2/20 brd 172.16.31.255 scope global dynamic br0
      valid_lft 79299sec preferred_lft 79299sec
   inet6 2001:660:4406:100:9::2/128 scope global 
      valid_lft forever preferred_lft forever
   inet6 fe80::9abe:94ff:fe68:7234/64 scope link 
      valid_lft forever preferred_lft forever

You can also check that your node has auto-configured a default route for IPv6:

Terminal.png node:
ip -6 route show default

It should give something similar to: 

default via fe80::204:96ff:fe97:b403 dev br0 proto ra metric 1024 expires 1711sec hoplimit 64 pref medium

The gateway will be different depending on the site.

In a deployed environment

Deployed environments typically don't have a network bridge configured (br0). You have to use the network interface directly, for instance:

Terminal.png node:
dhclient -6 INTERFACE_NAME
Note.png Note

You must find out the name of the network interface. Use ip link show and/or see the cluster hardware description such as Grenoble:Hardware#dahu to find the interface names of nodes. Mind that the interface name can change depending on the deployed environment (different OS, Linux Distribution, or version).

IPv6 communication from a grid5000 node to the Internet

Once your node has obtained an IPv6 address, it should be able to communicate with any host on the Internet:

Terminal.png node:
ping6 www.geant.net
Terminal.png node:
traceroute6 www.geant.net
Terminal.png node:

While outbound access to the Internet is not restricted, all connections are logged, see FAQ#Is_access_to_the_Internet_possible_from_nodes.3F.

IPv6 communication from the Internet to a grid5000 node

By default and as in most platforms, inbound communication from the Internet to a node is not allowed, the Grid'5000 network is protected by a firewall.

This firewall can be opened by the users with the Reconfigurable Firewall service. Users can request the firewall to be opened for some protocols / some ports, for nodes of their jobs. See the Reconfigurable Firewall documentation.

IPv6 communication with other testbeds

Fed4FIRE VLAN stitching

Several testbeds from Fed4FIRE can be interconnected with Grid'5000 using layer-2 VLAN stitching. More details here: Fed4FIRE VLAN Stitching.

Once this is done, you can use IPv6 on top of the stitched VLANs, with a usage that is similar to running IPv6 in a Kavlan.

External testbeds

Other testbeds are considered part of the Internet. As such, the general considerations for external traffic apply:

  • the other testbed needs to support IPv6
  • you can make outbound IPv6 communication from Grid'5000
  • inbound communication from the other testbed can be allowed with the Reconfigurable Firewall.

If the other testbed filters inbound IPv6 traffic, you may need to request it to allow traffic from the Grid'5000 IPv6 prefix (2001:660:4406::/48), or more selectively from the IPv6 addresses of your Grid'5000 nodes (this is out of the control of Grid'5000 tools or technical team).

IoT-LAB

IoT-LAB is currently an external testbed from the point of view of the Grid'5000 network. It supports IPv6 and does not filter inbound IPv6 traffic.

You can currently run experiments involving IoT-LAB in the following way:

SILECS

In the future, testbeds that are part of SILECS are expected to be directly interconnected with the current Grid'5000 sites. As such, they will no longer be considered "external".

IPv6 communication between grid5000 nodes

Nodes within Grid'5000 can communicate over IPv6 with each other.

DNS names for IPv6

To facilitate communication, all nodes have DNS names for IPv6:

Terminal.png node:
ping6 dahu-2-ipv6.grenoble.grid5000.fr

DNS names are different for IPv4 and IPv6: dahu-2.grenoble.grid5000.fr (IPv4) vs. dahu-2-ipv6.grenoble.grid5000.fr (IPv6).

Communication matrix on a single site

Two nodes on the same site can communicate in the following way:

Two nodes

in same site

Production network Local VLAN Routed VLAN Global VLAN
Production network Yes

(switched)

No Yes

(routed)

Yes

(routed)

Local VLAN No Yes*

(switched)

No No
Routed VLAN Yes

(routed)

No Yes

(switched or routed)

Yes

(routed)

Global VLAN Yes

(routed)

No Yes

(routed)

Yes

(switched or routed)

*Local VLAN: the nodes can only communicate if they are in the same local VLAN.

switched: the nodes can reach each other directly at layer 2, which allows features such as multicast.

routed: communication goes through a router. Performance should be similar to the switched case.

Communication matrix between sites

Two nodes from different sites can communicate in the following way:

Two nodes

on different sites

Production network Local VLAN Routed VLAN Global VLAN
Production network Yes

(routed)

No Yes

(routed)

Yes

(routed)

Local VLAN No No No No
Routed VLAN Yes

(routed)

No Yes

(routed)

Yes

(routed)

Global VLAN Yes

(routed)

No Yes

(routed)

Yes

(switched or routed)


switched: the nodes can reach each other directly at layer 2, which allows features such as multicast.

routed: communication goes through one or more routers. Performance should be similar to the switched case.

Advanced IPv6 usage in Grid'5000

IPv6 in kavlan

Note.png Note

The content below only covers IPv6-specific behaviour. For a general introduction on Kavlan, see KaVLAN

IPv6 is supported in kavlan-based VLANs.

IPv6 address configuration

Using DHCPv6

Once your node is in a VLAN, you can ask for an IPv6 address using a DHCPv6 client:

Terminal.png node:
dhclient -6 YOUR_INTERFACE_NAME
Note.png Note

This will not work in a local kavlan: you need to configure addresses statically in local kavlans, see below.

Of course, you need to be able to access your nodes to run this command. Here are several ways to do it:

  • if your node already has an IPv4 address in the kavlan, you can connect with SSH over IPv4:
Terminal.png frontend:
ssh root@grisou-39-kavlan-5 dhclient -6 YOUR_INTERFACE_NAME
Static address configuration

This is necessary for nodes that are in a local kavlan, because we don't provide any DHCPv6 service in this kind of VLAN. But you may also decide to configure addresses statically in other cases (routed VLAN, global VLAN).

To configure an IPv6 address statically:

Terminal.png node:
ip addr add 2001:660:4406:XXXX:YYYY:ZZZZ:TTTT/64 dev YOUR_INTERFACE_NAME

To find out the correct IPv6 address assigned to your node, you can look up the DNS entries from any machine connected to the Grid'5000 network:

Terminal.png frontend:
nslookup dahu-15-kavlan-3-ipv6.grenoble.grid5000.fr

or equivalently:

Terminal.png frontend:
dig +short dahu-15-kavlan-3-ipv6.grenoble.grid5000.fr AAAA

For more advanced usage, refer to the #Addressing_plan to pick a prefix that makes sense depending on the site and on the kavlan VLAN ID.

Connecting to nodes over IPv6

First, you need to make sure that your nodes have obtained an IPv6 address in the kavlan, see above.

Secondly, you must connect from a machine that has IPv6 connectivity. If your lab or home network provides IPv6, you are good to go. Otherwise, you need to find a third-party machine with IPv6 connectivity. One way is to reserve a Grid'5000 node and enable IPv6 on it (using dhclient -6 br0).

Routed and global VLAN

Use the IPv6 DNS names:

Terminal.png yourhost:
ssh root@grisou-39-kavlan-5-ipv6

where 5 is the kavlan ID accessible with kavlan -V.

Local VLAN

In a local VLAN, you can use an "SSH jump" through the kavlan server:

Terminal.png yourhost:
ssh -J kavlan1 root@grisou-39-kavlan-1-ipv6

where 1 is the kavlan ID accessible with kavlan -V.

Disabling the DHCPv6 server

You can disable the DHCPv6 server with the kavlan command (add -j JOBID if needed) :

Terminal.png frontend:
kavlan -d
Note.png Note

This command will disable both the DHCPv4 and DHCPv6 servers.

To enable the DHCP server again: kavlan -e.

Nodes with multiple network interfaces

Note.png Note

The content below only covers IPv6-specific behaviour. For a general introduction on how to use multiple network interfaces, see Advanced_KaVLAN#A_simple_multi_NICs_example

Nodes with multiple network interfaces can use IPv6, with the following important caveat: activating a network interface will automatically create an IPv6 default route, except in a local kavlan. This is because Grid'5000 routers advertise a default route using Router Advertisement packets, and most Linux system take these packets into account by default.

If you are not careful when experimenting, you can end up with unexpected default routes and even multiple default routes!

Disabling the automatic default route

If you don't want to automatically receive a default route on an interface, but still want to use DHCPv6 to configure IPv6 addresses, you can disable just the default route on this interface:

Terminal.png node:
sysctl net.ipv6.conf.YOUR_INTERFACE_NAME.accept_ra_defrtr=0

or the equivalent command which may be easier to script:

Terminal.png node:
echo 0 > /proc/sys/net/ipv6/conf/YOUR_INTERFACE_NAME/accept_ra_defrtr

This is useful because Router Advertisements configure two different kind of routes on the nodes:

  • default IPv6 route
  • local routes to other hosts in the same network

The above configuration only disables the first kind of routes.

Warning.png Warning

There is a sysctl setting net.ipv6.conf.all.accept_ra_defrtr that would appear to configure all interfaces, but it does not actually work this way. You need to apply this setting to each interface you want to configure.

Disabling all automatic routes

To completely disable all automatic routes, including local routes to other hosts in the same network:

Terminal.png node:
sysctl net.ipv6.conf.YOUR_INTERFACE_NAME.accept_ra=0

In that case, you will need to configure IPv6 addresses statically on your interface.

Putting network interfaces in different VLANs

See KaVLAN#Setting_up_the_VLAN

After changing the VLAN associated to an interface, it is recommended to flush old routes and addresses on this interface:

Terminal.png node:
ip -6 route flush dev YOUR_INTERFACE_NAME
Terminal.png node:
ip -6 address flush dev YOUR_INTERFACE_NAME scope global

Note the scope global option that avoids flushing the link-local IPv6 address.

Configuring IPv6 addresses on network interfaces

See #IPv6_in_kavlan

Current status

  • Grid'5000 has a global unicast IPv6 prefix (routable on the Internet).
  • An IPv6 address plan is defined in this prefix, where all interfaces of all nodes have a global unicast address (compared to IPv4, there is no NAT (Network Address Translation), no private addresses, all addresses are public, routable on the Internet). It is possible to connect from the Internet to Grid'5000 nodes directly, without the need to set up a tunnel or a VPN, using the Reconfigurable Firewall.
  • IPv6 routing is active between Grid'5000 sites, and with the Internet, same as IPv4.
  • IPv6 network traffic to and from the Internet goes through the two Grid'5000 Renater connections, in Lille or Sophia.
  • Traffic initiated from inside Grid'5000 is not filtered to the outside (Internet).
  • Traffic initiated from outside Grid'5000 (Internet) is currently filtered and blocked.
  • A dynamic firewall is provided. It allows users to open the firewall, for the duration of a job, to the node of the job. See Reconfigurable Firewall
  • Frontends currently don't have IPv6 addresses.
  • Grid'5000 internally provides DHCPv6 services to configure global unicast node addresses, and associated DNS / reverse DNS IPv6 records.
  • External DNS provides DNS / reverse DNS IPv6 records too.
  • IPv6 global unicast addresses are not configured by default on nodes' interfaces, since it can have unanticipated side-effects which can cause trouble to existing experiments or unaware users.
  • Nodes can get global unicast IPv6 addresses by issuing DHCPv6 requests.
  • Node interfaces have autoconfigured IPv6 link-local addresses, which is standard on Linux. This should not have any annoying side-effects.
  • IPv6 DNS records are in the form <node name>-ipv6.<site>.grid5000.fr. It was chosen to have different DNS names for IPv4 and IPv6 to avoid any DNS resolution confusion and because currently some Grid'5000 services (such as kadeploy) would be broken if the same DNS record resolves to both IPv4 and IPv6 (bug #10958, bug #10955).
  • Most Grid'5000 services are offered on IPv4 only. For example, Grid'5000 DNS servers do provide IPv4 and IPv6 records, but they only operate on IPv4.

Additional details

  • IPv6 addresses of nodes are in the reference api, under the ip6 key (in the network_adapters section, alongside the IPv4 address).
  • The detailed DNS naming scheme is <node name>[-<interface>][-kavlan-<kavlan number>][-ipv6].<site>.grid5000.fr. The name without interface is always an alias (CNAME) for the name with the default interface. For example, the IPv6 address of the default interface of node grisou-3, in Nancy, in kavlan 4 is grisou-3-kavlan-4-ipv6.nancy.grid5000.fr, which is a CNAME of grisou-3-eth0-kavlan-4-ipv6.nancy.grid5000.fr (since the default interface on this node is eth0).
  • IPv6 is supported in routed kavlans (kavlans number 4 to 9) and global kavlans (kavlans number >= 10) in the same way it is in the default production vlan.
  • Currently we chose to deactivate DHCPv6 in local non-routed kavlans, contrary to IPv4. Because in IPv6, DHCPv6 is designed to work together with Network Discovery Protocol (NDP) Router Advertisements (RA). As these kavlans are not routed, we opted to not broadcast Router Advertisements on these kavlans and DHCPv6 ends up behaving counter-intuitively: It actually depends on the DHCPv6 client used: dhclient gets and assigns an address to the interface, but the routing table is not correctly configured. dhcpcd by default refuses to assign the address (bug #12487). Anyway, anyone doing IPv6 in a local non routed kavlans will probably want to control IPv6 addresses assignment explicitly.
  • When switching an interface on the fly from a vlan to another vlan (with kavlan), the node is not aware of the change and it will by default remember old routes. Switching vlans on the fly is probably quite rare as most users will deploy in the kavlans, and kadeploy switches the vlan just before rebooting in the deployed environment, so this issue does not occur (bug #12477).
  • If a node has multiple interfaces in different vlans, each interface may receive Router Advertisements, which will add routes through different vlans. This may cause weird behaviors (bug #12486). If you want to activate different interfaces in different vlans, you have to know what you're doing and make sure your routing tables are consistent.
  • Allocation of addresses on several interfaces may not work in some situations due to the default behaviour of dhclient. When running interactively dhclient -6 on a node, the first time, on an interface, it correctly gets its IPv6. But when running again dhclient -6 on another interface, it fails due to DAD (duplicate address detection). The cause is that by default dhclient creates a lease file at the first run, in which it stores the DUID (DHCP Unique Identifier) of the client. Next requests will reuse this DUID, the DHCP server sees this, and so tries to allocate the same IPv6 to the other secondary interface. It is possible to prevent this for example by removing the lease file or telling dhclient to use an alternate lease file.
  • By default nodes and most servers will have a IPv6 link-local address and a default route (given by router advertisement). This should not cause any issues in most cases because if applications use the default behavior of getaddrinfo() (configurable through /etc/gai.conf), there won't be such problem since IPv4 is preferred when the only IPv6 address configured is a link-local. But some applications may assume IPv6 connectivity works since there is an IPv6 address and a default route (bug #11901: isc-bind behaves like this).

Addressing plan

Global prefix

Renater has allocated this /48 for Grid5000: 2001:0660:4406::/48

Overview

A B C D E F G H
Address,

groups of four hex digits:

2001 0660 4406 xxxx xxxx xxxx xxxx xxxx
What? 48 bits given by renater

fixed

16 bits available for

our subnetting needs

64 bits interface address
Purpose: routing (outside Grid5000) routing (inside Grid5000) identify an interface
Remark: part of our addressing plan

for routing purpose

part of our addressing plan

for organizational purpose

The D part from most significant to least significant

8 most significant bits: global Grid'5000 networks and sites

Network Range Subnet Range
backbone 2001:660:4406:0::/56
sites 2001:660:4406:100::/56
to
2001:660:4406:df00::/56
(223 possibles sites)
 grenoble 2001:660:4406:100::/56
lille 2001:660:4406:200::/56
luxembourg 2001:660:4406:300::/56
lyon 2001:660:4406:400::/56
nancy 2001:660:4406:500::/56
nantes 2001:660:4406:600::/56
rennes 2001:660:4406:700::/56
sophia 2001:660:4406:800::/56
kavlan-NG 2001:660:4406:e000::/52
(4096 simultaneous possible kavlans)
DMZ 2001:660:4406:f000::/52
(16 possible DMZs) 		DMZ south 2001:660:4406:fe00::/56
DMZ north 2001:660:4406:ff00::/56

(Part of this is configured globally in input/grid5000/ipv6.yaml)

Note.png Note

These site indexes are not the same as the ones from the IPv4 address plan (the 4 most significant bits of the third byte of the IPv4 address.

8 least significant bits: more detailed subnetting when applicable

This more detailed subnetting is not used at all for kavlan NG network, and is only partially applicable for backbone or DMZ networks.

network value subnetting details, remarks
production network 2001:660:4406:XX00::/64
admin network 2001:660:4406:XX10::/64
high performance network 2001:660:4406:XX20::/64
virtual network 2001:660:4406:XX40::/64 currently not implemented
backbone interconnection 2001:660:4406:XX50::/64
kavlans 2001:660:4406:XX80::/58
(64 possibilities per site: 80 to bf) 		2001:660:4406:XX80::/60
local non-routed
(16 possibilities per site: 80 to 8f)

8_least_significant_bits_of_D_part = 0x80 + kavlan id - 1

  • local non-routed kavlan-1: 2001:660:4406:XX80::/64
  • local non-routed kavlan-2: 2001:660:4406:XX81::/64
  • local non-routed kavlan-3: 2001:660:4406:XX82::/64

XX = site index

2001:660:4406:XX90::/60
local-routed
(16 possibilities per site: 90 to 9f)

8_least_significant_bits_of_D_part = 0x90 + kavlan id - 4

  • local routed kavlan-4: 2001:660:4406:XX90::/64
  • local routed kavlan-5: 2001:660:4406:XX91::/64
  • local routed kavlan-6: 2001:660:4406:XX92::/64
  • local routed kavlan-7: 2001:660:4406:XX93::/64
  • local routed kavlan-8: 2001:660:4406:XX94::/64
  • local routed kavlan-9: 2001:660:4406:XX95::/64

XX = site index

2001:660:4406:XXa0::/64
global
(1 global kavlan per site)

No relationship between value and the kavlan id.
In global kavlans, all traffic is routed to the gateway of the kavlan's site. The 8 most significant bits of the D part of the address identify this site, and in a given global kavlan, all nodes of Grid5000 have the same IPv6 /64 prefix. But we still need to set different addresses to nodes, so in the case of global kavlans, we use the 8 most significant bits of the E part to encode the site.

  • global kavlan-11 (Grenoble): 2001:660:4406:01a0::/64
  • global kavlan-12 (Lille): 2001:660:4406:02a0::/64
  • global kavlan-13 (Lyon): 2001:660:4406:04a0::/64
  • global kavlan-14 (Nancy): 2001:660:4406:05a0::/64
  • global kavlan-16 (Rennes): 2001:660:4406:07a0::/64
  • global kavlan-18 (Sophia): 2001:660:4406:08a0::/64
  • global kavlan-20 (Luxembourg): 2001:660:4406:03a0::/64
  • global kavlan-21 (Nantes): 2001:660:4406:06a0::/64
2001:660:4406:XXc0::/58 currently unused

The interface part

For production, admin, high performance, kavlan networks

  • E: server / cluster index:
    • General case:
      • 0 for servers or network equipments
      • 1..n for clusters (2^16-1 clusters possible per site). We take as cluster index the same index as in IPv4, except that we add 1, because this IPv6 address plan starts numbering clusters at 1, whereas the IPv4 address plan starts numbering clusters at 0. Script to retrieve automatically these IPv6 cluster indexes (with the +1 already included) (it simply takes the 4 low bits of the third octet of the IPv4 address of a node of the cluster and adds 1): https://gitlab.inria.fr/mimbert/g5k-ipv6-tools/blob/master/cluster_indexes
    • Global kavlan case:
      • 8 Most significant bits: site index
      • 8 least significant bits: as in the general case
      • Examples:
        • node taurus-1.lyon in global kavlan of Lyon (kavlan-13): 2001:660:4406:4a0:401::1
        • node taurus-1.lyon in global kavlan of Luxembourg (kavlan-20): 2001:660:4406:3a0:401::1
        • node chetemi-1.lille in global kavlan of Lyon (kavlan-13): 2001:660:4406:4a0:206::1
        • node chetemi-1.lille in global kavlan of Luxembourg (kavlan-20): 2001:660:4406:3a0:206::1
  • F:
    • for servers, type of server:
      • 0 network equipments
      • 1 physical servers
      • 2 virtual machines
    • for nodes: node interface, with 0 for the main/default interface. 2^16-1 possible interfaces per node.
  • G,H:
    • for network equipments: FFFF:FFFF for gateway. network equipment index otherwise
    • for servers: server index
    • for nodes: node index, starting from 1. 2^32-2 nodes possible per cluster.

A few examples:

  • The gateway of Rennes Admin network: 2001:660:4406:710::ffff:ffff
  • Node sagittaire-12.lyon.grid5000.fr: 2001:660:4406:400:2::c
Retrieved from "https://www.grid5000.fr/mediawiki/index.php?title=IPv6&oldid=103296"