Partitions
The default partition is LONG, which has a default of 24 hours of wallclock time per job and a maximum of 48 hours. There is a higher priority partition called TEST which allows up to four hours per job and has a dedicated X5650 compute node. Jobs put into TEST will normally run immediately. For longer jobs there are XLONG, which allows up to one week of wallclock time but is limited to a maximum of 96 cores, and XXLONG which allows up to 30 days but is limited to a maximum of 56 cores and only permits single node jobs.
Number of cores and nodes should be specified on the command line, or in the job script file.
Here are some examples to get started. You can combine most of the options.
# set wall time limit to 48 hours sbatch -t 48:00:00 myscript
# use the XLONG partition for up to 168 hours walltime sbatch -p XLONG -t 168:00:00 myscript
# 48 cores anywhere they fit sbatch -n 48 myscript
# 192 cores anywhere they fit sbatch -n 192 myscript
# 16 cores on a single E5-2650 node (good for MPI) sbatch -C new -n 16 -N 1 myscript
# 32 cores with 16 on each of two E5-5650 nodes (good for MPI) sbatch -C new -n 16 -N 2 myscript
# one 10-threaded SMP process on a X5650 node # Need to also set OMI_NUM_THREADS or the equivalent to 10 in the script sbatch -C old -n 1 -c 10 myscript
# 12 cores on one node in the test queue (faster turnaround but max 4 hours walltime) sbatch -p TEST -n 12 -N 1
# 4 cores on one node in the test queue sbatch -p TEST -n 4 -N 1
One node (currently node-0-0, an X5650 node) is permanently assigned to the test partition to provide quick access to CPUs for short jobs, and won't run any jobs out of other partitions. However if any other nodes are free and the system has more test jobs that fit onto node-0-0 at once then the scheduler may run test jobs on some of the other nodes.
Parallel work
Cerebro is a cluster system and so message passing libraries are used to make code run in parallel. Shared memory parallelization (OpenMP, provided by the autoparallelizing options on some of the compilers) can also be used for up to sixteen cores on E5-2650 nodes and twelve cores on X5650 nodes but not further.
The MPI libraries are all OpenMPI. You need to match your MPI library to your compiler if you're using Fortran so there are different libraries for different compilers.
To change your parallel environment you need to load the appropriate module. To load a module permanently you'd edit your .bashrc file and put in a module add line. If you want to use different libraries in different jobs then use the module commands inside your job script to get the environment right before launching. You might need to 'source /etc/profile.d/modules.sh' at the top of the script.
Compile your code with the MPI compilers. These are called mpicc, mpicxx, mpif77 and mpif90.
To run MPI code you normally use a command such as mpirun to launch your program. On cerebro there is a queueing system which assigns work to compute nodes, and so the MPI library has to interact with that which makes things more complicated. You use a special MPI launcher program to start your job. The launcher doesn't take any arguments but works out the correct number of cpus and which nodes to use from the queueing system. You must use the appropriate one for your MPI library, and you must have your environment configured for the library you want to use when you submit a job, as otherwise the job won't be able to find its library files. Here's a list of which launcher command goes with which library:
MPI | Command name |
---|---|
OpenMPI | mpirun |