Sophia:Network: Difference between revisions
Jump to navigation
Jump to search
No edit summary |
No edit summary |
||
| Line 11: | Line 11: | ||
{{:Sophia:GeneratedNetwork}} | {{:Sophia:GeneratedNetwork}} | ||
== High Performance networks == | == High Performance networks == | ||
| Line 39: | Line 20: | ||
==== Nodes ==== | ==== Nodes ==== | ||
*<code class="host">uvb- | *<code class="host">uvb-9</code> to <code class="host">uvb-12</code> have one QDR Infiniband card. | ||
* Card Model : Mellanox Technologies MT26428 [ConnectX IB QDR, PCIe 2.0 5GT/s]. | * Card Model : Mellanox Technologies MT26428 [ConnectX IB QDR, PCIe 2.0 5GT/s]. | ||
* Driver : <code class="dir">mlx4_ib</code> | * Driver : <code class="dir">mlx4_ib</code> | ||
* OAR property : ib_rate=40 | * OAR property : ib_rate=40 | ||
* IP over IB addressing : | * IP over IB addressing : | ||
** <code class="host">uvb-[ | ** <code class="host">uvb-[9..12]-ib0</code>.sophia.grid5000.fr ( 172.18.132.[1..44] ) | ||
==== Interconnection ==== | ==== Interconnection ==== | ||
| Line 55: | Line 32: | ||
Infiniband network is physically isolated from Ethernet networks. Therefore, Ethernet network emulated over Infiniband is isolated as well. There isn't any interconnexion, neither at the data link layer nor at the network layer. | Infiniband network is physically isolated from Ethernet networks. Therefore, Ethernet network emulated over Infiniband is isolated as well. There isn't any interconnexion, neither at the data link layer nor at the network layer. | ||
== Architecture == | == Rack Architecture == | ||
[[Image:2025-03-17-Urbanisation-Racks-Abaca%2BNef-dcSAM.png|center|thumb|view of the Baie|600px]] | [[Image:2025-03-17-Urbanisation-Racks-Abaca%2BNef-dcSAM.png|center|thumb|view of the Baie|600px]] | ||
Revision as of 14:18, 17 March 2025
Network: Global | Grenoble | Lille | Louvain | Luxembourg | Lyon | Nancy | Nantes | Rennes | Sophia | Strasbourg | Toulouse
See also: Hardware description for Sophia
Overview of Ethernet network topology
Network devices models
- gw: Aruba 8325-48Y8C
- sw-2: Aruba 6300M
- sw-3: Aruba 6300M
- sw-5: Aruba 2930F
- sw-6: Aruba 2930F
- swadmin-2: Aruba 2930F
- swadmin-3: Aruba 2930F
- swadmin-4: Aruba 2930F
- swadmin-5: Aruba 2930F
- swadmin-6: Aruba 2930F
More details (including address ranges) are available from the Grid5000:Network page.
High Performance networks
Infiniband 40G on uvb
uvb cluster nodes are all connected to 40G infiniband switches. Since these two clusters are shared with the Nef procution cluster at INRIA Sophia, we are using Infiniband partitions to isolate the nodes from nef when they are available on grid5000. The partition dedicated to grid5000 is 0x8100. The ipoib interfaces on nodes are therefore named ib0.8100 instead of ib0.
To use the native openib driver of openmpi, you must set: btl_openib_pkey = 0x8100
Nodes
uvb-9touvb-12have one QDR Infiniband card.- Card Model : Mellanox Technologies MT26428 [ConnectX IB QDR, PCIe 2.0 5GT/s].
- Driver :
mlx4_ib - OAR property : ib_rate=40
- IP over IB addressing :
uvb-[9..12]-ib0.sophia.grid5000.fr ( 172.18.132.[1..44] )
Interconnection
Infiniband network is physically isolated from Ethernet networks. Therefore, Ethernet network emulated over Infiniband is isolated as well. There isn't any interconnexion, neither at the data link layer nor at the network layer.
