Grenoble:Network: Difference between revisions

From Grid5000
Jump to navigation Jump to search
No edit summary
No edit summary
Line 11: Line 11:
= Omni-Path Topology =
= Omni-Path Topology =


Each node of the dahu, yeti and lama clusters are connected to 100Gbps Dell Omni-Path switches (some of these switch are shared with GriCad, the Grenoble mesocenter).  
Nodes of the dahu, yeti and lama clusters are connected to Omni-Path switches (Intel Omni-Path 100Gbps), aside the Ethernet network.
 
This Omni-Path network interconnects both the Grid'5000 Grenoble nodes and the nodes of the HPC Center of Université Grenoble Alpes (Gricad mésocentre).
 
The topology used is a fat tree with a 2:1 blocking factor:
* 2 top switchs
* 6 leaves : switchs with 32 downlinks to nodes and 8 uplinks to each top switch (48 ports total)
 
All 32 dahu nodes are connected to a same leave. The 4 yeti nodes are connected to another leave. TBC for lama.


One switch has 48 ports, 32 are reserved for the nodes. The 16 others ports are used to connect to two top switch. This is a fat tree blocking 2:1 topologoy (we have 16 uplinks for 32 nodes).


If we take 2 switch and the 2 top switch :
If we take 2 switch and the 2 top switch :

Revision as of 14:56, 2 December 2019

Overview of Ethernet network topology

GrenobleNetwork.png


Network devices models

  • gw: Dell S5296F-ON

More details (including address ranges) are available from the Grid5000:Network page.

Omni-Path Topology

Nodes of the dahu, yeti and lama clusters are connected to Omni-Path switches (Intel Omni-Path 100Gbps), aside the Ethernet network.

This Omni-Path network interconnects both the Grid'5000 Grenoble nodes and the nodes of the HPC Center of Université Grenoble Alpes (Gricad mésocentre).

The topology used is a fat tree with a 2:1 blocking factor:

  • 2 top switchs
  • 6 leaves : switchs with 32 downlinks to nodes and 8 uplinks to each top switch (48 ports total)

All 32 dahu nodes are connected to a same leave. The 4 yeti nodes are connected to another leave. TBC for lama.


If we take 2 switch and the 2 top switch :

                                                        +-----------------------------------------------------+               +-----------------------------------------------------+
                                                        |    OPA Top switch 1  (48 ports)                     |               |    OPA Top switch 2  (48 ports)                     |
                                                        +-----------------------------------------------------+               +-----------------------------------------------------+
                                                        |8 ports |8 p     |8 p     |8 p     |8 p     |8 p     |               |8 ports |8 p     |8 p     |8 p     |8 p     |8 p     |
                                                        +---+X+-------X---------------------------------------+               +---+X+-------X---------------------------------------+
                                                             X        X                                                            X        X
                                                             X        X                                                            X        X
                                                             X        X                                                            X        X
                                                    8 links  X        XXXXXXXXXXXXXXX                                              X        X
                                                             X                      X                                              X        XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
                                                             X                      X                                              X                                                                                 X
                                                             XXXXXXXXXXX            X                                              X                                                                                 X
                                                                       X            X                                              X                                                                                 X
                                                                       X            X 8 links                              8 links X                                                                                 X
                                                                       X            X                                              X                                                                        8 links  X
            +-----------------------------------------------------+    X            X                                              X                                                                                 X
            |    OPA switch 1  (48 ports)                         |    X            X                                              X                                                                                 X
            +-----------------------------------+--------+--------+    X            X                                              X                                                                                 X
            | 32 ports                          |        |        |    X            X                                              X                                                                                 X
            ++--+---------------+------------+--+---+X+--+--+X+---+    X            X                                              X                                                                                 X
             |  |               |            |       X       X         X            X                                              X                                                                                 X
             |  |               |            |       X       X         X            X                                              X                                                                                 X
             |  |               |            |       XXXXXXXXXXXXXXXXXXX            X                                              X                                                                                 X
      +------+  |               |            |               X                      X                                              X                                                                                 X
      |         |               |            |               X                      X                                              X                                                                                 X
      |         |               |            |               XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX                                                                                 X
      |         |            +--+---+        |                                      X                                                                                                                                X
      |         | +------+   |dahu N|  +-----+-+                                    X                                                                                                                                X
   +--+---+     +-+dahu 2|   +------+  |dahu 32|                                    X                                                                    +-----------------------------------------------------+     X
   |dahu 1|       +------+             +-------+                                    X                                                                    |    OPA switch 2  (48 ports)                         |     X
   +------+                                                                         X                                                                    +-----------------------------------+--------+--------+     X
                                                                                    X                                                                    | 32 ports                          |        |        |     X
                                                                                    X                                                                    ++--+---------------+------------+--+---+X+--+--+X+---+     X
                                                                                    X                                                                     |  |               |            |       X       X          X
                                                                                    X                                                                     |  |               |            |       X       X          X
                                                                                    X                                                                     |  |               |            |       X       X          X
                                                                                    X                                                              +------+  |               |            |       X       XXXXXXXXXXXXX
                                                                                    X                                                              |         |               |            |       X
                                                                                    X                                                              |         |               |            |       X
                                                                                    X                                                              |         |            +--+---+        |       X
                                                                                    X                                                              |         | +------+   |yeti 3|  +-----+-+     X
                                                                                    X                                                           +--+---+     +-+yeti 2|   +------+  |yeti 4 |     X
                                                                                    X                                                           |yeti 1|       +------+             +-------+     X
                                                                                    X                                                           +------+                                          X
                                                                                    X                                                                                                             X
                                                                                    X                                                                                                             X
                                                                                    XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

IP Range

  • Computing : 172.16.16.0/20
  • Omni-Path : 172.18.16.0/20
  • Virtual : 10.132.0.0/14