skip to content
 

Batch jobs

These are run by writing a script and submitting it to the queue with the sbatch command like this:

sbatch myscript

Scripts for batch jobs must start with the interpreter to be used to excute them (different from PBS/Torque). You can give arguments to sbatch as comments in the script. Example:

#!/bin/bash
# Name of the job
#SBATCH -J testjob
# Partition to use - this example is commented out
##SBATCH -p NONIB
# Time limit. Often not needed as there are defaults, 
# but you might have to specify it to get the maximum allowed.
# time: 10 hours
##SBATCH --time=10:0:0
# Pick nodes with feature 'foo'. Different clusters have 
# different features available
# but most of the time you don't need this
##SBATCH -C foo
# Restrict the job to run on the node(s) named
##SBATCH -w compute-0-13
# Number of processes
#SBATCH -n1
# Start the program
srun myprogram

A more complicated example which uses three tasks, note the use of --exact to make sure each job step only asks for the resources specifically allocated to it:

#!/bin/bash
#SBATCH -n3 # 3 tasks
echo Starting job $SLURM_JOB_ID
echo SLURM assigned me these nodes
srun -l hostname
srun --exact -n2 program1 & # start 2 copies of program 1
srun --exact -n1 program2 & # start 1 copy of program 2
wait # wait for all to finish

You should find a detailed example script in /info/slurm on the cluster you are using.

Interactive jobs

These can be run in two ways, via salloc and srun. If you just want a single interactive session on a compute node then using srun to allocate resources for a single task and launch a shell as that one task is probably the way to go. But if you want to run things in parallel or more than one task at once in your interactive job, use salloc to allocate resources and then srun or mpirun to start the tasks, as starting multiple copies of an interactive shell at once probably isn't what you want.

# One interactive task. Quit the shell to finish
srun --pty -u bash -i
# One task with one GPU assigned (GPU clusters only, obviously)
srun --pty --gpus=1 -u bash -i 
# One task with one maxwell GPU
srun --pty --gpus=1 -C maxwell -u bash -i 
# one task with one Titan Black GPU 
srun --pty --gpus=1 -C titanblack -u bash -i 
# One task with one GPU on the node called 'happy'
srun --pty --gpus=1 -w happy -u bash -i 
# Allocate three tasks, followed by running three instances of 'myprog' within the allocation.
# Then start one copy of longprog and two copies of myprog2, then release the allocation
salloc -n3
srun myprog
srun -n1 --exact longprog &
srun -n2 --exact myprog2 
exit

If you are using interactive srun to experiment with setting up jobs you should add -W 0 to the command line. Otherwise SLURM helpfully kills your interactive job when something fails or you pause too long between launching tasks.

Useful commands

squeue # view the queue

scancel <jobid> # cancel a job

sinfo # See the state of the system

sacct -l -j <jobid> # List accounting info about a job

Asking for resources

 

salloc/srun/sbatch support a huge array of options which let you ask for nodes, cpus, tasks, sockets, threads, memory etc. If you combine them SLURM will try to work out a sensible allocation, so for example if you ask for 13 tasks and 5 nodes SLURM will cope. Here are the ones that are most likely to be useful:

Option Meaning
-n Number of tasks (roughly, processes)
-N Number of nodes to assign. If you're using this, you might also be interested in --tasks-per-node
--tasks-per-node Maximum tasks to assign per node if using -N
--cpus-per-task Assign tasks containing more than one CPU. Useful for jobs with shared memory parallelization
-C Features the nodes assigned must have
-w Names of nodes that must be included - for selecting a particular node or nodes
--gpus=X Ask for X GPUs in the job. It's expected to use all X GPUs in a single job step.
--gpus=per-task=1 -nX Ask for X tasks, each with one GPU. This one is useful for jobs which want to assign several GPUs and then launch individual job steps on those GPUs. The steps can be launched with srun -n1 -N1 --exact .

 

MPI jobs

Inside a batch script you should just be able to call mpirun, which will communicate with SLURM and launch the job over the appropriate set of nodes for you:

#!/bin/bash
# 13 tasks over 5 nodes
#SBATCH -n13 -N5
echo Hosts are
srun -l hostname
mpirun /home/cen1001/src/mpi_hello

To run MPI jobs interactively you can assign some nodes using salloc, and then call mpirun from inside that allocation. Unlike PBS/Torque, the shell you launch with salloc runs on the same machine you ran salloc on, not on the first node of the allocation. But mpirun will do the right thing.

salloc -n12 bash
mpirun /home/cen1001/src/mpi_hello

You can even use srun to launch MPI jobs interactively without mpirun's intervention. The --mpi option here is to tell srun which method the MPI library uses for launching tasks. This is the correct one for use with our OpenMPI installations. However using srun like this isn't normally necessary unless you want to bind MPI processes to particular CPUs or sockets. Although mpirun has options for binding, they can interact badly with SLURM.

srun --mpi=pmi2 -n13 ./mpi_hello

Non-MPI Parallel jobs

In a parallel job which doesn't use MPI you can find out which hosts you have and how many by running "srun -l hostname" inside the job script. The -l option will print the slurm task number next to the assigned hostname for the task, skip it if you want just the list of hostnames.

You can then use "srun --exclusive" inside the job to start individual tasks, spreading them out over the available resources.

File copying in parallel jobs

If you want to run commands to set up files on every node in your job then this idiom is helpful

srun -n $SLURM_NNODES --tasks-per-node=1 mkdir /scratch/${SLURM_JOB_USER}/${SLURM_JOB_ID}
srun -n $SLURM_NNODES --tasks-per-node=1 cp file1 file2 /scratch/${SLURM_JOB_USER}/${SLURM_JOB_ID}

There is also an sbcast command that copies files to every node in the job, but that's all it does. It can't set up directories or delete files.

Jobs with multiple GPUs on GPU clusters

To get GPUs you can use --gpus=X, but in that case it's expected you'll use all the GPUs in a single job step (ie you launch one program that uses them all at once). To assign several GPUs and then allocate these to different tasks use --gpus-per-task and then srun to launch tasks within the job:

#!/bin/bash
#SBATCH -J testmultigpu
#SBATCH --gpus-per-task=1
#SBATCH -n4
# show us what resources we have: these run over everything
srun -l hostname
srun -l echo $CUDA_VISIBLE_DEVICES

# assign one instance of show.sh to each GPU over all four GPUs.
# Have to set -N here or it will default to the number of nodes in the whole job and print a warning
srun --exact -l -n1 -N1 /home/cen1001/show.sh &
srun --exact -l -n1 -N1 /home/cen1001/show.sh &
srun --exact -l -n1 -N1 /home/cen1001/show.sh &
srun --exact -l -n1 -N1 /home/cen1001/show.sh &
wait

System status 

System monitoring page

Can't find what you're looking for?

Then you might find our A-Z site index useful. Or, you can search the site using the box at the top of the page, or by clicking here.