IPv6

IPv6 basics

• On the node, choose the network interface on which you want to activate IPv6. For example, the main address on the standard environment is always br0.
• You can view the status of the interfaces with ip address show or more shortly ip a.
• Notice that br0 already has an autoconfigured link local inet6 address. These addresses are of the form fe80::/10. It's normal, these addresses are not routed.
• Notice also that IPv6 routes are already configured: ip -6 route. But it's not possible to communicate with other nodes or the outside world with IPv6 since there is only a link local address.
• To get the global unicast address of the interface, issue a DHCPv6 request on this interface. For this, one needs to be root. This can be done by deploying the node or with sudo-g5k. For example:

$ ip a
$ sudo-g5k dhclient -6 br0
$ ip a

• Now <br0> has a second inet6 address beginning with 2001:660:4406 which is the global unicast IPv6 prefix of Grid'5000.
• It's now possible to communicate with another node (if the other node also has an IPv6 routable address) or the outside world:

$ ping6 google.com
$ traceroute6 google.com
$ curl -6 https://tools.ietf.org/rfc/rfc2464.txt

• You can check the IPv6 DNS record of the node:

$ host $(ip a | sed -ne 's#.*\(2001:660:4406:[^/]*\).*#\1#p')

Current status

• Grid'5000 has a global unicast IPv6 prefix (routable on the Internet).
• An IPv6 address plan is defined in this prefix, where all interfaces of all nodes have a global unicast address (compared to IPv4, there is no NAT (Network Address Translation), no private addresses, all addresses are public, routable on the Internet). It will soon be possible to connect from the Internet to Grid'5000 nodes directly, without the need to set up a tunnel or a VPN.
• IPv6 routing is active between Grid'5000 sites, and with the Internet, same as IPv4.
• IPv6 network traffic to and from the Internet goes through the two Grid'5000 POPs, in Nancy or Sophia.
• Traffic initiated from inside Grid'5000 is not filtered to the outside (Internet).
• Traffic initiated from outside Grid'5000 (Internet) is currently filtered and blocked.
• A dynamic firewall is in development. It will allow users to open the firewall, for the duration of a job, to the node of the job (bug #11923).
• Grid'5000 internally provides DHCPv6 services to configure global unicast node addresses, and associated DNS / reverse DNS IPv6 records.
• External DNS will soon provide DNS / reverse DNS IPv6 records too (bug #12506).
• IPv6 global unicast addresses are not configured by default on nodes' interfaces, since it can have unanticipated side-effects which can cause trouble to existing experiments or unaware users.
• Nodes can get global unicast IPv6 addresses by issuing DHCPv6 requests.
• Node interfaces have autoconfigured IPv6 link-local addresses. This should not have any annoying side-effects.
• IPv6 DNS records are in the form <node name>-ipv6.<site>.grid5000.fr. It was chosen to have different DNS names for IPv4 and IPv6 to avoid any DNS resolution confusion and because currently some Grid'5000 services (such as kadeploy) would be broken if the same DNS record resolves to both IPv4 and IPv6 (bug #10958, bug #10955).
• Most Grid'5000 services are offered on IPv4 only. For example, Grid'5000 DNS servers do provide IPv4 and IPv6 records, but they only operate on IPv4.

Additional details

• IPv6 addresses of nodes are in the reference api, under the ip6 key (in the network_adapters section, alongside the IPv4 address).
• The detailed DNS naming scheme is <node name>[-<interface>][-kavlan-<kavlan number>][-ipv6].<site>.grid5000.fr. The name without interface is always an alias (CNAME) for the name with the default interface. For example, the IPv6 address of the default interface of node grisou-3, in Nancy, in kavlan 4 is grisou-3-kavlan-4-ipv6.nancy.grid5000.fr, which is a CNAME of grisou-3-eth0-kavlan-4-ipv6.nancy.grid5000.fr (since the default interface on this node is eth0).
• IPv6 is supported in routed kavlans (kavlans number 4 to 9) and global kavlans (kavlans number >= 10) in the same way it is in the default production vlan.
• Currently we chose to deactivate DHCPv6 in local non-routed kavlans, contrary to IPv4. Because in IPv6, DHCPv6 is designed to work together with Network Discovery Protocol (NDP) Router Advertisements (RA). As these kavlans are not routed, we opted to not broadcast Router Advertisements on these kavlans and DHCPv6 ends up behaving counter-intuitively: It actually depends on the DHCPv6 client used: dhclient gets and assigns an address to the interface, but the routing table is not correctly configured. dhcpcd by default refuses to assign the address (bug #12487). Anyway, anyone doing IPv6 in a local non routed kavlans will probably want to control IPv6 addresses assignment explicitly.
• When switching an interface on the fly from a vlan to another vlan (with kavlan), the node is not aware of the change and it will by default remember old routes. Switching vlans on the fly is probably quite rare as most users will deploy in the kavlans, and kadeploy switches the vlan just before rebooting in the deployed environment, so this issue does not occur (bug #12477).
• If a node has multiple interfaces in different vlans, each interface may receive Router Advertisements, which will add routes through different vlans. This may cause weird behaviors (bug #12486). If you want to activate different interfaces in different vlans, you have to know what you're doing and make sure your routing tables are consistent.
• Allocation of addresses on several interfaces may not work in some situations due to the default behaviour of dhclient. When running interactively dhclient -6 on a node, the first time, on an interface, it correctly gets its IPv6. But when running again dhclient -6 on another interface, it fails due to DAD (duplicate address detection). The cause is that by default dhclient creates a lease file at the first run, in which it stores the DUID (DHCP Unique Identifier) of the client. Next requests will reuse this DUID, the DHCP server sees this, and so tries to allocate the same IPv6 to the other secondary interface. It is possible to prevent this for example by removing the lease file or telling dhclient to use an alternate lease file.
• By default nodes and most servers will have a IPv6 link-local address and a default route (given by router advertisement). This should not cause any issues in most cases because if applications use the default behavior of getaddrinfo() (configurable through /etc/gai.conf), there won't be such problem since IPv4 is preferred when the only IPv6 address configured is a link-local. But some applications may assume IPv6 connectivity works since there is an IPv6 address and a default route (bug #11901: isc-bind behaves like this).

Address plan

Global prefix

Renater has allocated this /48 for Grid5000: 2001:0660:4406::/48

Overview

	A	B	C	D	E	F	G	H
Address, groups of four hex digits:	2001	0660	4406	xxxx	xxxx	xxxx	xxxx	xxxx
What?	48 bits given by renater fixed			16 bits available for our subnetting needs	64 bits interface address
Purpose:	routing (outside Grid5000)			routing (inside Grid5000)	identify an interface
Remark:				part of our addressing plan for routing purpose	part of our addressing plan for organizational purpose

The D part from most significant to least significant

8 most significant bits for the site (255 possible sites, the DMZ is one of them)

With special cases: backbone is zero, dmz start from the end (ff). Stored in input/grid5000/ipv6.yaml:

Site	Index	Prefix	Remark(s)
backbone	0	2001:660:4406:0000::/56	the backbone is a network rather than a site, so this may never be used.
grenoble	1	2001:660:4406:0100::/56
lille	2	2001:660:4406:0200::/56
luxembourg	3	2001:660:4406:0300::/56
lyon	4	2001:660:4406:0400::/56
nancy	5	2001:660:4406:0500::/56
nantes	6	2001:660:4406:0600::/56
rennes	7	2001:660:4406:0700::/56
sophia	8	2001:660:4406:0800::/56
dmz-north	255	2001:660:4406:ff00::/56
dmz-south	254	2001:660:4406:fe00::/56

	Note
	These site indexes are not the same as the ones from the IPv4 address plan (the 4 most significant bits of the third byte of the IPv4 address.

8 least significant bits: network (255 possible networks)

network	value	subnetting	details, remarks
production network	2001:660:4406:XX00::/64
admin network	2001:660:4406:XX10::/64
high performance network	2001:660:4406:XX20::/64
virtual network	2001:660:4406:XX40::/64		currently not implemented
backbone interconnection	2001:660:4406:XX50::/64
kavlans	2001:660:4406:XX80::/58 (64 possibilities per site: 80 to bf)	2001:660:4406:XX80::/60 local non-routed (16 possibilities per site: 80 to 8f)	8_least_significant_bits_of_D_part = 0x80 + kavlan id - 1 local non-routed kavlan-1: 2001:660:4406:XX80::/64 local non-routed kavlan-2: 2001:660:4406:XX81::/64 local non-routed kavlan-3: 2001:660:4406:XX82::/64 XX = site index
		2001:660:4406:XX90::/60 local-routed (16 possibilities per site: 90 to 9f)	8_least_significant_bits_of_D_part = 0x90 + kavlan id - 4 local routed kavlan-4: 2001:660:4406:XX90::/64 local routed kavlan-5: 2001:660:4406:XX91::/64 local routed kavlan-6: 2001:660:4406:XX92::/64 local routed kavlan-7: 2001:660:4406:XX93::/64 local routed kavlan-8: 2001:660:4406:XX94::/64 local routed kavlan-9: 2001:660:4406:XX95::/64 XX = site index
		2001:660:4406:XXa0::/64 global (1 global kavlan per site)	No relationship between value and the kavlan id. In global kavlans, all traffic is routed to the gateway of the kavlan's site. The 8 most significant bits of the D part of the address identify this site, and in a given global kavlan, all nodes of Grid5000 have the same IPv6 /64 prefix. But we still need to set different addresses to nodes, so in the case of global kavlans, we use the 8 most significant bits of the E part to encode the site. global kavlan-11 (Grenoble): 2001:660:4406:01a0::/64 global kavlan-12 (Lille): 2001:660:4406:02a0::/64 global kavlan-13 (Lyon): 2001:660:4406:04a0::/64 global kavlan-14 (Nancy): 2001:660:4406:05a0::/64 global kavlan-16 (Rennes): 2001:660:4406:07a0::/64 global kavlan-18 (Sophia): 2001:660:4406:08a0::/64 global kavlan-20 (Luxembourg): 2001:660:4406:03a0::/64 global kavlan-21 (Nantes): 2001:660:4406:06a0::/64
	2001:660:4406:XXc0::/58		currently unused

The interface part

For production, admin, high performance, kavlan networks

• E: server / cluster index:
  • General case:
    • 0 for servers or network equipments
    • 1..n for clusters (2^16-1 clusters possible per site). We take as cluster index the same index as in IPv4, except that we add 1, because this IPv6 address plan starts numbering clusters at 1, whereas the IPv4 address plan starts numbering clusters at 0. Script to retrieve automatically these IPv6 cluster indexes (with the +1 already included) (it simply takes the 4 low bits of the third octet of the IPv4 address of a node of the cluster and adds 1): https://gitlab.inria.fr/mimbert/g5k-ipv6-tools/blob/master/cluster_indexes
  • Global kavlan case:
    • 8 Most significant bits: site index
    • 8 least significant bits: as in the general case
    • Examples:
      • node taurus-1.lyon in global kavlan of Lyon (kavlan-13): 2001:660:4406:4a0:401::1
      • node taurus-1.lyon in global kavlan of Luxembourg (kavlan-20): 2001:660:4406:3a0:401::1
      • node chetemi-1.lille in global kavlan of Lyon (kavlan-13): 2001:660:4406:4a0:206::1
      • node chetemi-1.lille in global kavlan of Luxembourg (kavlan-20): 2001:660:4406:3a0:206::1
• F:
  • for servers, type of server:
    • 0 network equipments
    • 1 physical servers
    • 2 virtual machines
  • for nodes: node interface, with 0 for the main/default interface. 2^16-1 possible interfaces per node.
• G,H:
  • for network equipments: FFFF:FFFF for gateway. network equipment index otherwise
  • for servers: server index
  • for nodes: node index, starting from 1. 2^32-2 nodes possible per cluster.

A few examples:

• The gateway of Rennes Admin network: 2001:660:4406:710::ffff:ffff
• Node sagittaire-12.lyon.grid5000.fr: 2001:660:4406:400:2::c