Starting NAMD-Jobs on GPUs

NAMD allows the combination of OpenMPI and CUDA to start a job on several GPUs on several nodes.

NAMD

Loead required modules

module add namd/2.12-ibverbs-smp-cuda
module add openmpi/latest

NAMD works best with one thread per GPU. Using 2 or more GPUs on a node requires the combination of shared memory (thread based) and OpenMPI parallelisation, so called hybrid parallelisation. A threads can not saturate a GPU and therefore it is best to use all cores of a node.

GPU usage and hybrid parallelisation complicates job submission and prevents a formulation on a single line but requires scripting. Below you find a sample script called namd_skript.

To use 2 nodes with 16 cores and 2 GPUs of type V100 each, the script looks like:

#!/bin/bash			# required in first line
#SBATCH -t 60			# time limit set to 60 minutes
#SBATCH -p your_project 	# select a project or insert # in first column
#SBATCH --mail-type=END		# want a mail notification at end of job
#SBATCH -J jobname		# name of the job 
#SBATCH -o jobname.%j.out	# Output: %j expands to jobid
#SBATCH -e jobname.%j.err	# Error: %j expands to jobid
#SBATCH -L ompi			# request a license for openmpi
#SBATCH --nodes=2		# requesting 2 nodes (identical -N 2)
#SBATCH --ntasks-per-node=1     # 1 MPI task will be started per node
#SBATCH --cpus-per-task=16      # each MPI task starts 16 threads
#SBATCH --gres=gpu:V100:2	# use 2 GPUs of type V100 on each node

if [[ $# -eq 0 ]]; then
 echo "Aufruf: namd_skript <Eingabe Datei>"
 exit
fi

DEVICES=2
#Überschreibe das von LSF geschriebene Hostfile
chmod u+w $LSB_DJOB_HOSTFILE
echo "" > $LSB_DJOB_HOSTFILE   
i=1
for v in $LSB_MCPU_HOSTS; do
 if [ $i -eq 1 ]; then
   NODE=$v
   i=0
 else
   for j in `seq 1 $(($v/$DEVICES))`; do
     echo $NODE >> $LSB_DJOB_HOSTFILE
   done
   i=1
 fi
done
# Bestimme die Zahl der zu startenden SMP-Tasks
let LSB=$LSB_DJOB_NUMPROC/$DEVICES

# Starte den Job mit Eingabedatei als 1. Aufrufparameter
charmrun ++mpiexec ++remote-shell mpiexec $NAMD +p$LSB +devices 0,1 +idlepoll $1

---

This has not been tested. If something doesn't work - please inform us.