6. Compiling and Linking

Before compiling in the Tux environment, first ssh to one of the login nodes. Next, load those modules corresponding to the compiler, MPI version (if needed), and any third-party libraries required by your application. The load order should always be compiler first, MPI second, and third-party libraries last.

For example, suppose your application requires MPI and the HDF5 library. To compile using the Intel compiler and Intel MPI, the sequence of module commands would be:

ingves@tux:~$ module purge
ingves@tux:~$ module load Intel
ingves@tux:~$ module load hdf5

Supporting library-modules will be loaded as needed, and your environment will be updated so that the appropriate library directories are prepended to your $PATH and $LD_LIBRARY_PATH. The standard compiler variables FC, CC and CXX are set as appropriate for your compiler/MPI combination. These environment variables reference to the Fortran, C, and C++ compilers, respectively

In addition, several environment variables are set that may be useful during the compilation process. These variables are postfixed by ROOT and may easily be found by searching your environment for ROOT via env | grep ROOT. This will yield output similar to:

ingves@tux:~$ env | grep ROOT
I_MPI_ROOT=/share/apps/software/Core/intel/2021.4/mpi/2021.4.0
INTEL_ONEAPI_MPI_ROOT=/share/apps/software/Core/intel/2021.4
INTEL_ONEAPI_COMPILERS_ROOT=/share/apps/software/Core/intel/2021.4
INTEL_ONEAPI_MKL_ROOT=/share/apps/software/Core/intel/2021.4
MKLROOT=/share/apps/software/Core/intel/2021.4/mkl/2021.4.0
HDF5_ROOT=/share/apps/spack/v1.0.7/software/linux-debian11-sandybridge/intel-2021.4.0/hdf5-1.10.7-m24yqqdiqzoldlbek3ustkz65aeuc56v
[...]

Once the relevant modules are loaded, you are ready to compile. For our HDF5 example, a compilation command that uses the environment variables set by the module system may look like:

ingves@tux:~$ $FC my_program.f90 -I$HDF5_ROOT/inc -L$HDF5_ROOT/lib -lhdf5_fortran -o my_program

Note

Your run-time environment should reflect your compilation environment. Be sure to include the same sequence of module commands in your job script as that used at compile time. [This may not always be needed due to the use of RPATH, but it is still a recommended practice.]

6.1. Compiler and Optimization Recommendations

There are two main classes of compiler suites installed in tux

GNU Compiler Collection (gcc stack)
Intel oneAPI compilers (Intel stack)

The tux cluster runs on Intel hardware. As such, we strongly recommend using the Intel compiler along with Intel's MPI library when compiling software. For production, we suggest compiling with the -O2 or -O3 optimization flags along with the vectorization flags appropriate for the node you plan to run on. For Haswell nodes, this means compiling with the -xCORE-AVX2 flag. For the Xeon-Phi and Skylake nodes, use -xCORE-AVX512.

Compilation commands for a typical Tux Haswell node should resemble:

ingves@tux:~$ $FC -O3 -xCORE-AVX2 my_program.f90 -o my_program
ingves@tux:~$ $CC -O3 -xCORE-AVX2 my_program.c -o my_program
ingves@tux:~$ $CXX -O3 -xCORE-AVX2 my_program.cpp -o my_program

For the Skylake nodes, the appropriate commands would be:

ingves@tux:~$ $FC -O3 -xCORE-AVX512 my_program.f90 -o my_program
ingves@tux:~$ $CC -O3 -xCORE-AVX512 my_program.c -o my_program
ingves@tux:~$ $CXX -O3 -xCORE-AVX512 my_program.cpp -o my_program

6.2. Linking to the Math Kernel Library (MKL)

The Intel Math Kernel Library (MKL) provides optimized routines for a number of common mathematical operations. Notably, it provides interfaces to the LAPack and BLAS linear algebra libraries as well as the FFTW Fourier transform package.

If you wish to link MKL to your Intel-compiled application, use the -mkl flag:

ingves@tux:~$ $CXX -O3 -xCORE-AVX2 my_program.cpp -o my_program -mkl

If your application uses FFTW, you will also need to include MKL's FFTW directory in your compilation command:

ingves@tux:~$ $CXX -O3 -xCORE-AVX2 -I$MKL_ROOT/include/fftw my_program.cpp -o my_program -mkl

For the GNU and NVHPC compilers, the link syntax becomes more complex. The Intel Link Advisor can be used to generate the appropriate linking syntax based on your application's needs.

For the GNU compiler, linking against sequential MKL libraries, the appropriate Fortran linking syntax is:

ingves@tux:~$ $FC my_program.f90  -m64 -I$MKL_ROOT/include -o my_program.out  -L$MKL_ROOT/lib -Wl,--no-as-needed -lmkl_gf_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -ldl

The comparable c/c++ syntax would be:

ingves@tux:~$ $FC my_program.cpp  -m64 -I$MKL_ROOT/include -o my_program   -L$MKL_ROOT/lib -Wl,--no-as-needed -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -ldl

Note that if your application uses FFTW, you will must use the FFTW include flag just as with the Intel compiler. Consult the link advisor if you have additional questions about how to link MKL to your application.