Tux Quick Reference

Below some central tux cluster information is summarized for quick reference!

Partitions

Partition	Hosts	Time limit	#cores/node
short	all	1 hour	mixed
normal	tux-[1-9]	7 days	16
sandybridge	tux-[1-11]	30 days	16
haswell	tux-[12-14]	30 days	20
haswell	tux-[15-16]	30 days	24
skylake	tux-[17-24]	30 days	32
skylake	tux-[25-28]	30 days	40
cascadelake	tux-[29-30]	30 days	40

Filesystem

Name	Path	Availability	Backup	FS-type	Purpose
Home	/home/$USER	all nodes	Yes	NFS	Home-directory: user specific files, no calculation data.
SimLab	/simlab	all nodes	No	NFS	Project area for long-term-storage.
Work	/work/$USER	all nodes	No	NFS	Personal work (scratch) space for every user. Calculation data etc.
Scratch	/scratch	single node	No	ext4	Primary (and usually fastest) place for single-node calculation data. Removed once user's jobs are finished on the node.
tmp	/tmp	single node	No	tmpfs	Ramfs; fast in-memory filesystem

Module system

The main module commands:

Command (full)	Command (short)	Purpose
module list	ml	List active modules in the user environment
module avail [module]	ml av [module]	List available modules in MODULEPATH
module load <module>	ml <module>	Load a module file in the users environment
module unload <module>	ml rm <module>	Remove a loaded module from the user environment
module purge	ml purge	Remove all modules from the user environment
module swap <module1> <module2>	ml swap <module1> <module2>	Replace module1 with module2
module spider <module>	ml spider [module]	Query all modules in MODULEPATH and any module hierarchy
module show <module>	ml show <module>	Show content of commands performed by loading module file

Here <module> and [module] denotes required and optional module name, respectively.

The corresponding abbreviated shorthand module commands forms are:

Command	Abbreviation
module list	ml
module avail [module]	ml av [module]
module load <module>	ml <module>
module unload <module>	ml rm <module>
module purge	ml purge
module swap <module1> <module2>	ml swap <module1> <module2>
module spider <module>	ml spider [module]
module show <module>	ml show <module>

Slurm

Command	Functionality
sbatch	Submit a job script
salloc	Create an interactive SLURM shell.
srun	Execute argument command on the resources assigned to a job. Note: must be executed inside an active job (script or interactive environment); in most cases, using mpiexec (which in turn uses srun for startup) is the preferred alternative.
squeue	Print table of submitted jobs and their state. Note: non-privileged users can only see their own jobs.
sinfo	Provide overview of cluster status
scontrol	Query and modify SLURM state

Option	Short option	Description
--job-name=<JobName>	-J <JobName>	Job name in queue
--partition=<partition>	-p <partition>	Partition to use
--time=<D-HH:MM:SS>	-t <D-HH:MM:SS>	Maximum wall time
--nodes=<node>	-N <node>	# nodes to use
--ntasks=<tasks>	-n <tasks>	# tasks in total for the job
--ntasks-per-node=<tasks>		# tasks / node
--ntasks-per-core		# tasks / core
--cpus-per-task=<cores>	-c <cores>	# cores / task
--hint=[no]multithread		Hyperthreading Yes/No
--mem=<mem>		Memory / node
--mem-per-cpu=<mem>		Memory / core
--mail-type=<type>		Send email at start/end of job
--mail-user=<email>		Email address to use
--gres=<list>		Generic consumable resources (csv)
--constraint=<attribute>		Request certain features (e.g. bigmem)

Many more options are available for sbatch; to see them, inspect the manual page for the command or the relevant section of the Slurm homepage.

Variable	Description
SLURM_JOB_ID	Useful for naming output files that won't clash.
SLURM_JOB_NAME	Name of the job.
SLURM_SUBMIT_DIR	Directory where sbatch was called.
SLURM_ARRAY_TASK_ID	The current index of your array job.
SLURM_CPUS_PER_TASK	Useful as an input for multi-threaded functions.
SLURM_NTASKS	Useful as an input for MPI functions.

Common SLURM Environment Variables

Variable	Description
$SLURM_JOB_ID	The Job ID.
$SLURM_JOBID	Deprecated. Same as $SLURM_JOB_ID
$SLURM_SUBMIT_DIR	The path of the job submission directory.
$SLURM_SUBMIT_HOST	The hostname of the node used for job submission.
$SLURM_JOB_NODELIST	Contains the definition (list) of the nodes that is assigned to the job.
$SLURM_NODELIST	Deprecated. Same as SLURM_JOB_NODELIST.
$SLURM_CPUS_PER_TASK	Number of CPUs per task.
$SLURM_CPUS_ON_NODE	Number of CPUs on the allocated node.
$SLURM_JOB_CPUS_PER_NODE	Count of processors available to the job on this node.
$SLURM_CPUS_PER_GPU	Number of CPUs requested per allocated GPU.
$SLURM_MEM_PER_CPU	Memory per CPU. Same as --mem-per-cpu .
$SLURM_MEM_PER_GPU	Memory per GPU.
$SLURM_MEM_PER_NODE	Memory per node. Same as --mem .
$SLURM_GPUS	Number of GPUs requested.
$SLURM_NTASKS	Same as -n, --ntasks. The number of tasks.
$SLURM_NTASKS_PER_NODE	Number of tasks requested per node.
$SLURM_NTASKS_PER_SOCKET	Number of tasks requested per socket.
$SLURM_NTASKS_PER_CORE	Number of tasks requested per core.
$SLURM_NTASKS_PER_GPU	Number of tasks requested per GPU.
$SLURM_NPROCS	Same as -n, --ntasks. See $SLURM_NTASKS.
$SLURM_NNODES	Total number of nodes in the job's resource allocation.
$SLURM_TASKS_PER_NODE	Number of tasks to be initiated on each node.
$SLURM_ARRAY_JOB_ID	Job array's master job ID number.
$SLURM_ARRAY_TASK_ID	Job array ID (index) number.
$SLURM_ARRAY_TASK_COUNT	Total number of tasks in a job array.
$SLURM_ARRAY_TASK_MAX	Job array's maximum ID (index) number.
$SLURM_ARRAY_TASK_MIN	Job array's minimum ID (index) number.

A full list of environment variables for SLURM can be found by visiting the SLURM page on environment variables.

Hardware

Compute nodes

node	CPUs	Architecture	Freq [GHz]	Cores	Mem [GB]	IB	HT	scratch [GB]	Model	GPU
tux-1	Intel Xeon E5-2670 @ 2.60GHz	sandybridge	2.60	16	96	QDR	N	205	Dell PowerEdge R620	none
tux-2	Intel Xeon E5-2670 @ 2.60GHz	sandybridge	2.60	16	96	QDR	N	205	Dell PowerEdge R620	none
tux-3	Intel Xeon E5-2670 @ 2.60GHz	sandybridge	2.60	16	96	QDR	N	205	Dell PowerEdge R620	none
tux-4	Intel Xeon E5-2670 @ 2.60GHz	sandybridge	2.60	16	96	QDR	N	205	Dell PowerEdge R620	none
tux-5	Intel Xeon E5-2670 @ 2.60GHz	sandybridge	2.60	16	96	QDR	N	205	Dell PowerEdge R620	none
tux-6	Intel Xeon E5-2670 @ 2.60GHz	sandybridge	2.60	16	96	QDR	N	205	Dell PowerEdge R620	none
tux-7	Intel Xeon E5-2670 @ 2.60GHz	sandybridge	2.60	16	96	QDR	N	205	Dell PowerEdge R620	none
tux-8	Intel Xeon E5-2670 @ 2.60GHz	sandybridge	2.60	16	96	QDR	N	196	Dell PowerEdge R620	none
tux-9	Intel Xeon E5-2670 @ 2.60GHz	sandybridge	2.60	16	96	QDR	N	418	Dell PowerEdge R620	none
tux-10	Intel Xeon E5-2670 @ 2.60GHz	sandybridge	2.60	16	96	QDR	Y	437	Supermicro	none
tux-11	Intel Xeon E5-2670 @ 2.60GHz	sandybridge	2.60	16	96	QDR	Y	437	Supermicro	none
tux-12	Intel Xeon E5-2660 v3 @ 2.60GHz	haswell	2.60	20	128	FDR	Y	529	Dell PowerEdge R730	2 x NVIDIA GeForce GTX TITAN X
tux-13	Intel Xeon E5-2660 v3 @ 2.60GHz	haswell	2.60	20	128	FDR	Y	251	Dell PowerEdge R730	2 x NVIDIA GeForce GTX TITAN X
tux-14	Intel Xeon E5-2660 v3 @ 2.60GHz	haswell	2.60	20	128	FDR	Y	251	Dell PowerEdge R730	2 x NVIDIA GeForce GTX TITAN X
tux-15	Intel Xeon E5-2690 v3 @ 2.60GHz	haswell	2.60	24	128	FDR	Y	431	Dell PowerEdge R630	none
tux-16	Intel Xeon E5-2690 v3 @ 2.60GHz	haswell	2.60	24	128	FDR	Y	431	Dell PowerEdge R630	none
tux-17	Intel Xeon Gold 6142 @ 2.60GHz	skylake	2.60	32	196	FDR	N	211	HPE Proliant DL360 Gen10	none
tux-18	Intel Xeon Gold 6142 @ 2.60GHz	skylake	2.60	32	196	FDR	N	211	HPE Proliant DL360 Gen10	none
tux-19	Intel Xeon Gold 6142 @ 2.60GHz	skylake	2.60	32	196	FDR	N	211	HPE Proliant DL360 Gen10	none
tux-20	Intel Xeon Gold 6142 @ 2.60GHz	skylake	2.60	32	196	FDR	Y	418	HPE Proliant DL360 Gen10	none
tux-21	Intel Xeon Gold 6142 @ 2.60GHz	skylake	2.60	32	196	FDR	Y	418	HPE Proliant DL360 Gen10	none
tux-22	Intel Xeon Gold 6142 @ 2.60GHz	skylake	2.60	32	196	FDR	Y	418	HPE Proliant DL360 Gen10	none
tux-23	Intel Xeon Gold 6142 @ 2.60GHz	skylake	2.60	32	384	FDR	Y	418	HPE Proliant DL360 Gen10	none
tux-24	Intel Xeon Gold 6142 @ 2.60GHz	skylake	2.60	32	384	FDR	Y	418	HPE Proliant DL360 Gen10	none
tux-25	Intel Xeon Gold 6148 @ 2.40GHz	skylake	2.40	40	192	FDR	Y	418	Dell PowerEdge C6420	none
tux-26	Intel Xeon Gold 6148 @ 2.40GHz	skylake	2.40	40	128	FDR	Y	418	Dell PowerEdge C6420	none
tux-27	Intel Xeon Gold 6148 @ 2.40GHz	skylake	2.40	40	128	FDR	Y	418	Dell PowerEdge C6420	none
tux-28	Intel Xeon Gold 6148 @ 2.40GHz	skylake	2.40	40	128	FDR	Y	418	Dell PowerEdge C6420	none
tux-29	Intel Xeon Gold 5218R @ 2.10GHz	cascadelake	2.10	40	128	FDR	N	418	Dell PowerEdge R640	none
tux-30	Intel Xeon Gold 5218R @ 2.10GHz	cascadelake	2.10	40	128	FDR	N	418	Dell PowerEdge R640	none

Login and admin nodes

node	CPUs	Architecture	Freq [GHz]	Cores	Mem [GB]	IB	HT	scratch [GB]	Model	GPU
tuxlogin-0 / tux	Intel Xeon E5-2670 @ 2.60GHz	sandybridge	2.60	16	64	QDR	Y	247	Supermicro	none
tuxlogin-1	Intel Xeon E5-2670 @ 2.60GHz	sandybridge	2.60	16	128	FDR	Y	512	Dell PowerEdge	none
tuxnas	Intel Xeon E5-2630 v4 @ 2.20GHz	broadwell	2.20	20	64	FDR	Y	1800	Dell PowerEdge R730	none
tuxmaster	Intel Xeon E5-2690 v3 @ 2.60GHz	haswell	2.60	24	128	FDR	Y	none	Dell PowerEdge R630	none

Networking

Role	Brand	Model/MPN	IB Type	No. Ports	CA Type
IB-switch	Mellanox	MSX6025F-1SFS	FDR	36	---
IB-card	Mellanox	MCX354A-FCBT	FDR	2	MT4099
IB-card	Mellanox	MCX353A-FCBT	FDR	1	MT4099
IB-card	Mellanox	MT25408A0-FCC-QI	QDR	2	MT26428