Sophia:Network
IP networks in use
You have to use a public network range to run an experiment between several Grid5000 sites.
Public Networks
- computing : 138.96.20.0/22
- virtual : 10.164.0.0/14
Local Networks
- admin : 10.6.1.0/24
- ipmi : 10.96.20.0/24
Network
Gigabit Ethernet
Azur Cluster (IBM e325)
All the nodes are connected to 4 Cisco-3750 switches (gigabit ethernet). The switches are interconnected using a 32Gbps stack. The nodes are connected to the following switches:
- cisco1 (Master) : azur-1 to azur-15 (15 nodes), 10GB uplink
to the FastIron
- cisco2 : azur-16 to azur-36 (21 nodes)
- cisco3 : azur-37 to azur-55 (19 nodes)
- cisco4 : azur-56 to azur-72 (17 nodes)
The switches are dedicated to Grid5000 (since january 2007; before that, they were shared with the local cluster).
You can choose the switch for your reservation with OAR, using the switch name:
oarsub -l nodes=XX -p "switch='ciscoX'"
If you want XX nodes on a single switch, use
oarsub -l /switch=1/nodes=XX
Network usage is available through cricket at : https://helpdesk.grid5000.fr/cricket/grapher.cgi?target=%2Fsophia%2Fcisco
OS used:
Cisco IOS Software, C3750 Software (C3750-IPBASE-M), Version 12.2(25)SEB4
Helios Cluster (Sun X4100)
All the nodes are connected in gigabit ethernet to 2 Foudry EdgeIron switches (one Edgeiron 48GS and one EdgeIron24GS). The switches are interconnected using a 2x10Gbps stack (10GBps in each direction). The nodes are connected to the following switches:
- edgeiron48gs : helios01 to helios48 (48 nodes)
- edgeiron24gs : helios49 to helios56 (8 nodes)
You can choose the switch for your reservation with OAR, using the switch name: oarsub -l nodes=15 -p "switch='XXX'"
OS used: 2.2.6.35
Sol Cluster (Sun X2200 M2)
All the nodes are connected (in a non blocking way) to the main Foundry FastIron Super X switch.
47 nodes (from sol-1 to sol-47) have a second Ethernet interface connected to 2 Cisco 3750 stacked switches.
- IP addressing for eth1: sol-{1-47}-eth1
Topology
The main switch is a Foundry FastIron Super X. It has 2 dual 10GB modules, 2 modules with 24 gigabit ports, and 12 gigabit ports on the management module (so 60 gigabit ports are available). 4 slots are currently free.
The Cisco 3750 stack (Azur) is connected to the FastIron switch with a 10GB link; The EdgeIron stack (helios) is connected to the FastIron with 2 aggregated 10GB links.
Each frontend is connected to the FastIron switch.
Network usage is available through cricket at: https://helpdesk.grid5000.fr/cricket/grapher.cgi?target=%2Fsophia%2Fgw-sophia.sophia.grid5000.fr
IP pool for virtualization
Ongoing tests with Kavlan are using half of the virtualization address pool (see Virtual_network_interlink ). The parameters of available IPs are:
network 10.166.0.0/15 netmask 255.254.0.0 gateway 10.167.255.254
Myrinet
72 azur nodes and 56 helios nodes are connected to the 128 ports Myrinet 2000 switch.
You can submit jobs with OAR on Myrinet nodes using the 'myrinet' property: oarsub -l nodes=10 -p "myri2g='YES'".
MX driver is enabled by default on these nodes, IP emulation also.
- IP over MX addressing:
helios-{1-56}-myri0andazur-{1-72}-myri0(192.168.7.{1-128}) - Myrinet network usage is available through cricket at: https://helpdesk.grid5000.fr/cricket/