Sophia:Network: Difference between revisions

From Grid5000
Jump to navigation Jump to search
No edit summary
 
(78 intermediate revisions by 14 users not shown)
Line 1: Line 1:
= Réseau =
{{Template:Site link|Network}}
{{Portal|Network}}
{{Portal|User}}
 
'''See also:''' [[Sophia:Hardware|Hardware description for Sophia]]
 
= Overview of Ethernet network topology =
 
[[File:SophiaNetwork.svg|600px]]
 
{{:Sophia:GeneratedNetwork}}


== Gigabit Ethernet ==


=== Cluster Azur (IBM)===
= Old information (possibly outdated) =
Tous les noeuds du cluster IBM sont interconnectés en Gigabit Ethernet par une pile de 4 switchs Cisco-3750 reliés par un bus de 32Gbps. La liste précise le nombre de noeuds par switchs :


* switch n°5 : 19 noeuds
[[Image:Sophia_modified.jpg|center|thumb|Physical view of the compute network|600px]]
* switch n°6 : 19 noeuds
* switch n°8 : 21 noeuds
* '''switch n°9''' (Master) : 15 noeuds, lien 10G  vers FastIron


Cette pile réseau est maintenant dédié à Grid5000 (depuis janvier 2007).
== Gigabit Ethernet ==


Le numéro de switch peut être utilisé lors des réservations oar : <code class="command">oarsub -l nodes=15 -p switch=X</code>


Il est possible de consulter l'utilisation du réseau à l'adresse suivante : https://helpdesk.grid5000.fr/cricket/grapher.cgi?target=%2Fsophia%2Fcisco
=== Suno Cluster (Dell R410) ===


OS installé:
All the nodes are connected (in a non blocking way) to the main Foundry FastIron Super X switch.
<code>
Cisco IOS Software, C3750 Software (C3750-IPBASE-M), Version 12.2(25)SEB4</code>


=== Cluster Helios (Sun) ===
=== UVB Cluster ===


Tous les noeuds sont interconnectés en gigabit ethernet sur une pile de 2 switchs Foundry EdgeIron, reliés par un bus de 10Gbps (2x10 en full duplex). La liste précise l'appartenance des noeuds aux différents switchs :
All the nodes are connected to two stacked Dell PowerConnect 6248 switches. The Powerconnect stack is connected to the  Foundry FastIron Super X through a 10Gbps link.


* switch n°1 : helios01 à helios48 (48 noeuds)
=== Topology ===
* switch n°2 : helios49 à helios56 (8 noeuds)


Le numéro de switch peut être utilisé lors des réservations oar : <code class="command">oarsub -l nodes=15 -p switch=X</code>
The main switch is a Foundry FastIron Super X. It has 2 dual 10GB modules, 4 modules with 24 gigabit ports, and 12 gigabit ports on the management module (so 60 gigabit ports are available). 4 slots are currently free.


Version de l'OS: <code>2.2.6.23</code>
== High Performance networks ==


=== Interconnection ===
=== Infiniband 40G on uvb ===
uvb cluster nodes are all connected to 40G infiniband switches. Since these two clusters are shared with the Nef procution cluster at INRIA Sophia, we are using Infiniband partitions to isolate the nodes from nef when they are available on grid5000. The partition dedicated to grid5000 is '''0x8100'''. The ipoib interfaces on nodes are therefore named ib0.8100 instead of ib0.
To use the native openib driver of '''openmpi''', you must set: <code>btl_openib_pkey = 0x8100 </code>


L'interconnection entre les deux piles de switch se fait via deux liens 10G vers un commutateur Foundry FastIron Super X.
==== Nodes ====


L'interconnection Renater se fait via un port 10 Gigabit Ethernet sur le FastIron.
*<code class="host">uvb-1</code> to <code class="host">uvb-44</code> have one QDR Infiniband card.
* Card Model : Mellanox Technologies MT26428 [ConnectX IB QDR, PCIe 2.0 5GT/s].
* Driver : <code class="dir">mlx4_ib</code>
* OAR property : ib_rate=40
* IP over IB addressing :
** <code class="host">uvb-[1..44]-ib0</code>.sophia.grid5000.fr ( 172.18.132.[1..44] )


Il est possible de consulter l'utilisation du réseau à l'adresse suivante :
==== Switch ====
https://helpdesk.grid5000.fr/cricket/grapher.cgi?target=%2Fsophia%2Fgw-sophia.sophia.grid5000.fr


== Myrinet ==
* three Mellanox IS50xx QDR Infiniband Switchs
* Topology available here : https://wiki.inria.fr/ClustersSophia/Network (uvb are nef084-nef140 on the nef production cluster)


74 noeuds azur (tous) et 54 noeuds helios sont interconnectés via un commutateur Myrinet 2000 de 128 ports
==== Interconnection ====


Elles peuvent être réservée lors des soumissions oar en précisant la propriété 'myrinet' : <code class="command">oarsub -l nodes=10 -p "myrinet='yes'"</code>.
Infiniband network is physically isolated from Ethernet networks. Therefore, Ethernet network emulated over Infiniband is isolated as well. There isn't any interconnexion, neither at the data link layer nor at the network layer.


Il est possible de consulter l'utilisation du réseau à l'adresse suivante : https://helpdesk.grid5000.fr/cricket/
==  Architecture ==
[[Image:Sophia-cluster_modified.png|center|thumb|view of the Baie|600px]]

Latest revision as of 13:49, 12 March 2024

Network: Global | Grenoble | Lille | Luxembourg | Lyon | Nancy | Nantes | Rennes | Sophia | Strasbourg | Toulouse

See also: Hardware description for Sophia

Overview of Ethernet network topology

SophiaNetwork.svg


Network devices models

  • gw: Foundry
  • swx1nef: Juniper EX2500-24F-FB
  • swy1nef: Dell PowerConnect 6248

More details (including address ranges) are available from the Grid5000:Network page.


Old information (possibly outdated)

Physical view of the compute network

Gigabit Ethernet

Suno Cluster (Dell R410)

All the nodes are connected (in a non blocking way) to the main Foundry FastIron Super X switch.

UVB Cluster

All the nodes are connected to two stacked Dell PowerConnect 6248 switches. The Powerconnect stack is connected to the Foundry FastIron Super X through a 10Gbps link.

Topology

The main switch is a Foundry FastIron Super X. It has 2 dual 10GB modules, 4 modules with 24 gigabit ports, and 12 gigabit ports on the management module (so 60 gigabit ports are available). 4 slots are currently free.

High Performance networks

Infiniband 40G on uvb

uvb cluster nodes are all connected to 40G infiniband switches. Since these two clusters are shared with the Nef procution cluster at INRIA Sophia, we are using Infiniband partitions to isolate the nodes from nef when they are available on grid5000. The partition dedicated to grid5000 is 0x8100. The ipoib interfaces on nodes are therefore named ib0.8100 instead of ib0. To use the native openib driver of openmpi, you must set: btl_openib_pkey = 0x8100

Nodes

  • uvb-1 to uvb-44 have one QDR Infiniband card.
  • Card Model : Mellanox Technologies MT26428 [ConnectX IB QDR, PCIe 2.0 5GT/s].
  • Driver : mlx4_ib
  • OAR property : ib_rate=40
  • IP over IB addressing :
    • uvb-[1..44]-ib0.sophia.grid5000.fr ( 172.18.132.[1..44] )

Switch

Interconnection

Infiniband network is physically isolated from Ethernet networks. Therefore, Ethernet network emulated over Infiniband is isolated as well. There isn't any interconnexion, neither at the data link layer nor at the network layer.

Architecture

view of the Baie
Retrieved from "https://www.grid5000.fr/mediawiki/index.php?title=Sophia:Network&oldid=102310"