The short version
- Pick the executable:
vasp_stdfor normal runs,vasp_gamfor a single Γ point,vasp_nclfor noncollinear magnetism and spin–orbit coupling. - Set
ulimit -s unlimitedandOMP_NUM_THREADSin every script; most random segmentation faults are a stack that is too small. - Launch with the MPI that VASP was built with:
srunwhere the site supports it, never ampirunfrom another MPI. - KPAR must divide the number of MPI ranks; NCORE should divide the ranks per node. Set NCORE or NPAR, never both.
- On GPUs: one MPI rank per GPU, no NCORE (VASP forces 1), KPAR = number of GPUs if memory allows, a larger NSIM.
- Let Slurm warn the script before the time limit and write a STOPCAR, so VASP stops cleanly after an ionic step.
Write a job script
Describe the layout and the generator writes a Slurm script and the parallel tags for the INCAR. The script is generic: replace ACCOUNT, PARTITION and the module name with your site's values. It runs in your browser.
Templates
#!/bin/bash
# Replace ACCOUNT, PARTITION and the module name with your site's values.
#SBATCH -J relax
#SBATCH -A ACCOUNT
#SBATCH -p PARTITION
#SBATCH -N 2
#SBATCH --ntasks-per-node=64
#SBATCH -c 1
#SBATCH -t 24:00:00
#SBATCH -o %x-%j.out
#SBATCH -e %x-%j.err
module purge
module load vasp/VERSION
ulimit -s unlimited # VASP keeps large arrays on the stack
export OMP_NUM_THREADS=1
export OMP_STACKSIZE=512m
export OMP_PLACES=cores OMP_PROC_BIND=close
cd "$SLURM_SUBMIT_DIR" || exit 1
srun -n 128 -c 1 --cpu-bind=cores vasp_std > vasp.out#!/bin/bash
# Replace ACCOUNT, GPU_PARTITION and the module name with your site's values.
#SBATCH -J relax-gpu
#SBATCH -A ACCOUNT
#SBATCH -p GPU_PARTITION
#SBATCH -N 1
#SBATCH --gpus-per-node=4
#SBATCH --ntasks-per-node=4
#SBATCH -c 16
#SBATCH -t 12:00:00
#SBATCH -o %x-%j.out
#SBATCH -e %x-%j.err
module purge
module load vasp/VERSION-gpu
ulimit -s unlimited
export OMP_NUM_THREADS=16 # CPU-side threads per rank
export OMP_STACKSIZE=512m
export OMP_PLACES=cores OMP_PROC_BIND=close
cd "$SLURM_SUBMIT_DIR" || exit 1
srun -n 4 -c 16 --cpu-bind=cores --gpu-bind=none vasp_std > vasp.outINCAR for the GPU job: no NCORE, KPAR = 4 if the irreducible k-points divide evenly and memory allows, and NSIM well above 4 (benchmark 8 to 32).
Ranks, threads and the parallel tags
| Setting | Rule |
|---|---|
| Cores used | nodes × --ntasks-per-node × -c. Keep it at or below the physical cores: oversubscription slows everything. |
-c and threads | -c equals OMP_NUM_THREADS (twice that if the nodes count hyperthreads as CPUs). Pass -c to srun as well as to sbatch. |
| KPAR | Divides the MPI ranks and, ideally, the number of irreducible k-points (on GPUs, only when they divide evenly). Each k-point group holds its own copy of the data, so it costs memory. Small bulk cells with many k-points: large KPAR, NCORE = 1. VASP advice |
| NCORE | Ranks working on one band. A divisor of the ranks per node, near √(ranks per k-point group); the cores per NUMA domain is often best. About 4 for ~100 atoms, 12–16 above 400 atoms. Forced to 1 with OpenMP or on GPUs. VASP advice |
| NPAR | The old way to set the same thing (NPAR × NCORE = ranks per k-point group). If both are set, NPAR wins: delete it from old templates. |
| IMAGES (NEB) | The ranks are split over the images too: total ranks = IMAGES × KPAR × (ranks per k-point group). |
VASP's own recipe: do a dry run (vasp_std --dry-run, VASP 6.4 and later) to get NBANDS and the number of irreducible k-points, benchmark a few NCORE and KPAR values on a few SCF steps of the real system, then keep the fastest. The layout VASP actually used is at the top of OUTCAR.
| Tag | What it does | Default | What to use | Watch out |
|---|---|---|---|---|
NCORE | Number of cores that work together on one band. | 1 | Cores per NUMA domain is a good start; about 4 for ~100 atoms, 12–16 above 400 atoms. Benchmark. | Forced to 1 with OpenMP threads and on GPUs. Do not set NPAR as well. |
NPAR | Legacy: the number of band groups (MPI ranks ÷ NCORE). | the number of MPI ranks | Use NCORE instead. | If both are set, NPAR wins. |
KPAR | Number of k-point groups that run in parallel. | 1 | A divisor of the MPI ranks, up to the number of irreducible k-points if memory allows; on GPUs often the number of GPUs. | Each group holds its own copy of the data: memory is not shared. |
NSIM | Number of bands optimized at once in the blocked algorithms. | 4 | Higher on GPUs (up to 32). |
Which executable
| Executable | Use it for |
|---|---|
vasp_std | Every collinear run, spin-polarized or not, with a k-point mesh. The default. |
vasp_gam | Runs with only the Γ point: molecules, large supercells, most AIMD. |
vasp_ncl | Noncollinear magnetism and spin–orbit coupling (LNONCOLLINEAR, LSORBIT). vasp_std stops on such an INCAR. |
Many structures and chained jobs
#!/bin/bash
# ... the #SBATCH lines of a template, plus:
#SBATCH -a 0-199%20
#SBATCH -o logs/%x-%A_%a.out
DIR=$(sed -n "$((SLURM_ARRAY_TASK_ID + 1))p" dirs.txt)
cd "$DIR" || exit 1
srun vasp_std > vasp.out0-199%20 runs 200 tasks, at most 20 at a time. Create the log folder first (mkdir -p logs): Slurm does not, and tasks without it fail without a log.
jid1=$(sbatch --parsable relax.sh)
jid2=$(sbatch --parsable --dependency=afterok:$jid1 static.sh)
sbatch --dependency=afterok:$jid2 dos.shafterok checks only the exit code, and VASP exits normally after NSW steps even when the relaxation did not converge. Start the next script with a guard:
grep -q "reached required accuracy" ../relax/OUTCAR || { echo "relaxation not converged"; exit 1; }
cp ../relax/CONTCAR POSCARStop cleanly before the time limit
A job killed at the time limit can leave a half-written WAVECAR. A STOPCAR with LSTOP = .TRUE. stops VASP after the current ionic step; LABORT = .TRUE. stops at the next electronic step, leaving unconverged WAVECAR and CHGCAR, so prefer LSTOP. Slurm can send the script a signal a set time before the limit; the script then writes the STOPCAR.
#!/bin/bash
#SBATCH --signal=B:USR1@900
# ... the other #SBATCH lines; leave enough time (here 15 min) for about three ionic steps
rm -f STOPCAR # a leftover STOPCAR would stop the run at once
trap 'echo "LSTOP = .TRUE." > STOPCAR' USR1
srun vasp_std > vasp.out &
pid=$!
wait $pid; status=$?
if [ $status -gt 128 ]; then # the signal interrupted wait: let VASP finish its step
wait $pid; status=$?
fi
exit $status # keeps the exit code for --dependency=afterokSize the warning from the time per ionic step: grep LOOP+ OUTCAR prints it. To continue: keep OUTCAR, OSZICAR and vasprun.xml under new names, cp CONTCAR POSCAR, keep WAVECAR and CHGCAR (and TAUCAR for meta-GGA), resubmit.
Watch and manage jobs
squeue -u $USER # your jobs; the REASON column explains pending ones
scontrol show job JOBID # the full record: time limit, nodes, work directory
sacct -j JOBID --format=JobID,JobName%20,State,ExitCode,Elapsed,Timelimit,NNodes,NTasks,MaxRSS
seff JOBID # CPU and memory efficiency after the job ends (if installed)
scancel JOBID # cancel one job; scancel -u $USER cancels all yours
sinfo -s # partitions and how many nodes are freeIn sacct, State TIMEOUT means the time limit, OUT_OF_MEMORY an out-of-memory kill, NODE_FAIL a node problem. MaxRSS is recorded on job steps, so leave out -X when you want memory. Progress of a running job: tail -f vasp.out, or grep -c " F= " OSZICAR for the ionic steps done.
Common mistakes
- No
ulimit -s unlimited: random segmentation faults. - A
mpirunfrom a different MPI than VASP's: one rank with many threads, or a crash. - Ranks × threads above the physical cores, or
-cdifferent fromOMP_NUM_THREADS. - KPAR that does not divide the ranks; NCORE and NPAR both set.
- More than one MPI rank per GPU, or NCORE set on GPUs.
- Trusting
afterokas proof of convergence; a stale STOPCAR in the directory. - Several calculations in one directory, or running in a home directory instead of scratch.
- Spin-polarized r²SCAN on VASP 6.6.0, whose forces are wrong: check
head -1 OUTCAR. - With KPAR > 1 a run can hang in the SCF (a known issue, partly fixed in 6.6.0): if a job stalls, try KPAR = 1.
The physics behind this sheet
- Lecture 19: VASP input files and convergence tests (in preparation)
Further reading
- VASP Wiki, “Optimizing the parallelization” (NCORE, KPAR, the dry run and benchmarking). · www.vasp.at
- VASP Wiki, “GPU ports of VASP” (one MPI rank per GPU, NCORE, KPAR and NSIM on GPUs). · www.vasp.at
- VASP Wiki, “Combining MPI and OpenMP” (threads, stack size and pinning). · www.vasp.at
- Slurm documentation, “sbatch” (directives, --signal, --dependency and arrays). · slurm.schedmd.com