Network emulation

Introduction

The Grid'5000 network is built with high-performance network hardware and dedicated network links. This way, the infrastructure can support demanding experiments that make heavy use of network resources.

However, some scientific experiments may actually require a lower-performance network. This is especially the case for experiments that target lower-performance network environments, such as Edge / Fog computing, Peer-to-peer networks, or public blockchains. In other cases, an experimenter may want to vary network parameters (such as latency or packet loss) to study the impact of these parameters on application performance.

Grid'5000 offers a variety of resources in several geographic areas, which may be enough in some cases to build an experiment with the desired level of performance. If more control on network parameters is required, then network emulation can be used instead. Emulation allows to artificially degrade network performance to a desired level, while still using real hardware and a real network (unlike simulation).

The main network parameters that can be emulated are bandwidth, latency, and packet loss. Of course, these parameters can only be made worse than what the real network provides: if the base latency between two nodes is 5 milliseconds, emulation will not be able to provide a lower latency than this.

After reviewing simpler alternatives, this tutorial walks through two main methods to setup network emulation in Grid'5000: one using netem (the Linux NETwork EMulator) directly, and the other one using high-level tools such as distem and enoslib. These high-level tools add a layer of abstraction, but under the hood they also use netem to setup network emulation.

Alternatives to network emulation

Network emulation can be difficult to get right, so if there is a simpler way to obtain the required network parameters, it should be used first.

We describe two possible methods: how to obtain a specific link speed on a node, and how to obtain higher latency using multiple geographical sites.

Using nodes with a specific link speed

Nodes in Grid'5000 have network interfaces with different speed. For Ethernet, this is typically 1 Gbit/s, 10 Gbit/s, or 25 Gbit/s. See Hardware#Networking for a complete and up-to-date list.

Select a node with the desired interface speed, and if necessary follow the multi-NICs tutorial if the interface is not the first one.

In addition, it possible to force an interface to run at a lower speed: typically, force a 1 Gbit/s interface to run at 100 Mbit/s.

For instance, grisou nodes in Nancy have a secondary 1 Gbit/s interface called eno3. To set it up to 100 Mbit/s:

grisou$ sudo-g5k ethtool -s eno3 speed 100 duplex full autoneg off
grisou$ sudo-g5k ip link set eno3 up

You then have to make sure you are using this specific interface for your experiment: it means using KaVLAN if you want to isolate it, and configure appropriate routing.

Using two sites with high latency

Grid'5000 network

Grid'5000 has infrastructure in several geographical sites: see Grid5000:Network

As such, you can use nodes on two sites that are far apart, and this will result in natural network latency between groups of nodes. For instance, the latency between the sites of Nancy and Rennes is around 25 milliseconds.

There are several options to reserve nodes on different sites:

• simply reserve nodes on each site independently
• use OARGrid
• automate your experiment with scripting

By default, nodes in different sites are not in the same Ethernet network: traffic goes through backbone routers. If you need direct layer-2 connectivity between your nodes, you can use a global VLAN that is propagated to all Grid'5000 sites.

Basic network emulation with netem

Reserve nodes and use netem to emulate network conditions on it.

We use the production network for control, and apply netem on a secondary interface in a kavlan. This way, traffic to DNS servers, NFS, SSH, etc, will be unaffected by the emulated network.

Use-case: apply emulation directly on the nodes of your experiment. Insist on egress emulation (much easier to do than ingress emulation, and often enough).

Schéma

Reserve and deploy:

oarsub -t deploy -l "{type='kavlan'}/vlan=1 + {eth_count > 1}/nodes=2" -I

kadeploy3 -e debian10-x64-min -k -f $OAR_NODEFILE

Check access: ssh root@node-1 uptime

Determine name of a secondary interface on nodes (look at the node, or in the hardware page, or the API)

Move secondary interface to an isolated VLAN: kavlan -s -i $(kavlan -V) -m grimoire-1-eno2.nancy.grid5000.fr -m grimoire-4-eno2.nancy.grid5000.fr --verbose

Note: ethX can be used as an alias to the actual interface name.

Get an IP address on the secondary interface: ssh root@node-1 dhclient -v eno2

Optional: get an IPv6 address on the secondary interface (see IPv6 tutorial)

Check connectivity on kavlan: ssh root@node-1 ping node-2-eno2-kavlan-$(kavlan -V)

ssh root@node-1 tc qdisc replace dev eno2 root netem delay 10ms limit 10000 ssh root@node-2 tc qdisc replace dev eno2 root netem delay 10ms limit 10000

Test ping again: it should be around 20 ms (round-trip time, so it's 2x 10ms)

Example: test TCP performance under packet loss, with iperf3:

both# apt update && apt install -y iperf3 node2# iperf3 -s node1# iperf3 -c node-2-eno2-kavlan-XXX

No packet loss: you should get around 1.2 Gbit/s (note: iperf3 is not capable to saturate 10 Gbit/s)

Try again with a limit at 50 Mbit/s:

tc qdisc replace dev eno2 root netem delay 10ms rate 50Mbit limit 10000

You should get a bit less than 50 Mbit/s with iperf3.

Try again with 1% packet loss in forward direction:

ssh root@node-1 tc qdisc replace dev eno2 root netem delay 10ms limit 10000 loss 1%

Try again to run iperf3 from node1 to node2: you should now get around 10 Mbit/s (it's quite variable).

note on the "limit" parameter and how to choose it.

Emulation with netem on a physical node with Kavlan

Same idea, but physically separate networks.

Advanced network emulation tools

Deploy a topology in containers with distem or distrinet (based on mininet), which both support network emulation.

Automation with Enoslib

Enoslib is a generic solution where it is possible (in theory) to describe a topology and instantiate it on either Grid'5000 or distem.