Hardware details

The cluster consists of 30 compute nodes and 2 login nodes (sandybridge) in addition to a few service nodes. All nodes are dual-socket servers from the manufactures HPE, DELL and Supermicro. All nodes are equipped with Infiniband ConnectX-3 interconnects (FDR or QDR). Furthermore, three of the nodes have installed two GPUs (NVIDIA GeForce GTX TITAN X) each for calculations.

In total the cluster has 30 compute nodes that provides 780 physical compute cores and a total of 4.7TB of memory.

Compute nodes

node

CPUs

Architecture

Freq [GHz]

Cores

Mem [GB]

IB

HT

scratch [GB]

Model

GPU

tux-1

Intel Xeon E5-2670 @ 2.60GHz

sandybridge

2.60

16

96

QDR

N

205

Dell PowerEdge R620

none

tux-2

Intel Xeon E5-2670 @ 2.60GHz

sandybridge

2.60

16

96

QDR

N

205

Dell PowerEdge R620

none

tux-3

Intel Xeon E5-2670 @ 2.60GHz

sandybridge

2.60

16

96

QDR

N

205

Dell PowerEdge R620

none

tux-4

Intel Xeon E5-2670 @ 2.60GHz

sandybridge

2.60

16

96

QDR

N

205

Dell PowerEdge R620

none

tux-5

Intel Xeon E5-2670 @ 2.60GHz

sandybridge

2.60

16

96

QDR

N

205

Dell PowerEdge R620

none

tux-6

Intel Xeon E5-2670 @ 2.60GHz

sandybridge

2.60

16

96

QDR

N

205

Dell PowerEdge R620

none

tux-7

Intel Xeon E5-2670 @ 2.60GHz

sandybridge

2.60

16

96

QDR

N

205

Dell PowerEdge R620

none

tux-8

Intel Xeon E5-2670 @ 2.60GHz

sandybridge

2.60

16

96

QDR

N

196

Dell PowerEdge R620

none

tux-9

Intel Xeon E5-2670 @ 2.60GHz

sandybridge

2.60

16

96

QDR

N

418

Dell PowerEdge R620

none

tux-10

Intel Xeon E5-2670 @ 2.60GHz

sandybridge

2.60

16

96

QDR

Y

437

Supermicro

none

tux-11

Intel Xeon E5-2670 @ 2.60GHz

sandybridge

2.60

16

96

QDR

Y

437

Supermicro

none

tux-12

Intel Xeon E5-2660 v3 @ 2.60GHz

haswell

2.60

20

128

FDR

Y

529

Dell PowerEdge R730

2 x NVIDIA GeForce GTX TITAN X

tux-13

Intel Xeon E5-2660 v3 @ 2.60GHz

haswell

2.60

20

128

FDR

Y

251

Dell PowerEdge R730

2 x NVIDIA GeForce GTX TITAN X

tux-14

Intel Xeon E5-2660 v3 @ 2.60GHz

haswell

2.60

20

128

FDR

Y

251

Dell PowerEdge R730

2 x NVIDIA GeForce GTX TITAN X

tux-15

Intel Xeon E5-2690 v3 @ 2.60GHz

haswell

2.60

24

128

FDR

Y

431

Dell PowerEdge R630

none

tux-16

Intel Xeon E5-2690 v3 @ 2.60GHz

haswell

2.60

24

128

FDR

Y

431

Dell PowerEdge R630

none

tux-17

Intel Xeon Gold 6142 @ 2.60GHz

skylake

2.60

32

196

FDR

N

211

HPE Proliant DL360 Gen10

none

tux-18

Intel Xeon Gold 6142 @ 2.60GHz

skylake

2.60

32

196

FDR

N

211

HPE Proliant DL360 Gen10

none

tux-19

Intel Xeon Gold 6142 @ 2.60GHz

skylake

2.60

32

196

FDR

N

211

HPE Proliant DL360 Gen10

none

tux-20

Intel Xeon Gold 6142 @ 2.60GHz

skylake

2.60

32

196

FDR

Y

418

HPE Proliant DL360 Gen10

none

tux-21

Intel Xeon Gold 6142 @ 2.60GHz

skylake

2.60

32

196

FDR

Y

418

HPE Proliant DL360 Gen10

none

tux-22

Intel Xeon Gold 6142 @ 2.60GHz

skylake

2.60

32

196

FDR

Y

418

HPE Proliant DL360 Gen10

none

tux-23

Intel Xeon Gold 6142 @ 2.60GHz

skylake

2.60

32

384

FDR

Y

418

HPE Proliant DL360 Gen10

none

tux-24

Intel Xeon Gold 6142 @ 2.60GHz

skylake

2.60

32

384

FDR

Y

418

HPE Proliant DL360 Gen10

none

tux-25

Intel Xeon Gold 6148 @ 2.40GHz

skylake

2.40

40

192

FDR

Y

418

Dell PowerEdge C6420

none

tux-26

Intel Xeon Gold 6148 @ 2.40GHz

skylake

2.40

40

128

FDR

Y

418

Dell PowerEdge C6420

none

tux-27

Intel Xeon Gold 6148 @ 2.40GHz

skylake

2.40

40

128

FDR

Y

418

Dell PowerEdge C6420

none

tux-28

Intel Xeon Gold 6148 @ 2.40GHz

skylake

2.40

40

128

FDR

Y

418

Dell PowerEdge C6420

none

tux-29

Intel Xeon Gold 5218R @ 2.10GHz

cascadelake

2.10

40

128

FDR

N

418

Dell PowerEdge R640

none

tux-30

Intel Xeon Gold 5218R @ 2.10GHz

cascadelake

2.10

40

128

FDR

N

418

Dell PowerEdge R640

none

Login and Admin nodes

node

CPUs

Architecture

Freq [GHz]

Cores

Mem [GB]

IB

HT

scratch [GB]

Model

GPU

tuxlogin-0 / tux

Intel Xeon E5-2670 @ 2.60GHz

sandybridge

2.60

16

64

QDR

Y

247

Supermicro

none

tuxlogin-1

Intel Xeon E5-2670 @ 2.60GHz

sandybridge

2.60

16

128

FDR

Y

512

Dell PowerEdge

none

tuxnas

Intel Xeon E5-2630 v4 @ 2.20GHz

broadwell

2.20

20

64

FDR

Y

1800

Dell PowerEdge R730

none

tuxmaster

Intel Xeon E5-2690 v3 @ 2.60GHz

haswell

2.60

24

128

FDR

Y

none

Dell PowerEdge R630

none

Networking

For networking, the cluster relies on a Mellanox SX6025 InfiniBand switch, model MSX6025F-1SFS. This is a passive 36-port SwitchX-2 FDR switch. Furthermore, all tux nodes are equipped with single or dual ports FDR or QDR InfiniBand cards.

Here is a list of the InfiniBand (IB) networking hardware we have:

Role

Brand

Model/MPN

IB Type

No. Ports

CA Type

IB-switch

Mellanox

MSX6025F-1SFS

FDR

36

---

IB-card

Mellanox

MCX354A-FCBT

FDR

2

MT4099

IB-card

Mellanox

MCX353A-FCBT

FDR

1

MT4099

IB-card

Mellanox

MT25408A0-FCC-QI

QDR

2

MT26428