Skip to content

Submitting Jobs

This page explains how to use our compute cluster to run jobs in parallel and speed up your analyses.

The compute cluster is managed by a scheduler, whose job is to make efficient use of the resources and ensure fairness in resource allocation.

Partitions

The scheduler divides the cluster up into blocks of similarly capable hardware, called partitions. The OOD cluster uses these to provide a very course-grained job priority based on expected runtime and to select hardware with GPUs.

The different partitions have different run-times and memory limits, when a task reaches these limits it will be terminated; also shorter queues take precedence over the longer ones. It is advantageous to provide the scheduler with as much information about your job's memory and time requirements.

To help the scheduler achieve this, you can specify a partition when using fsl_sub. Typing fsl_sub --help gives you a list of partitions available for use with descriptions of allowed run times and memory availability - more details are available on our partitions and fair share pages.

How to run tasks on the cluster

The main command for submitting jobs on the cluster is fsl_sub. Type the following to get the usage:

fsl_sub --help

Typing fsl_sub before the rest of your command will send the job to the cluster. For example, to run FSL's BET on the cluster:

fsl_sub -N my_bet_job bet struct.nii.gz struct_brain.nii.gz

By default, fsl_sub will submit your task to the short partition (max 1.2day runtime). fsl_sub can automatically choose a partition for you if you provide information about your job's requirements. We would strongly recommend that you provide at least an estimated maximum run time (--jobtime).

For example, to queue a job which requires 10GiB of memory and runs for 2 hours use:

fsl_sub -R 10 -T 120 my_job

See Why do I need to specify RAM and Time? for more information, and How much RAM/Time does my job need to assist with setting these.

Warning

Before submitting any tasks make sure you have loaded any shell modules you require (e.g. for FSL, type module add fsl).

You can also submit a text file with a list of tasks that will then run in parallel. For example, to run BET on a list of subjects:

for subj in 001 002 003;do
    echo bet ${subj}/struct.nii.gz ${subj}/struct_brain.nii.gz >> commands.txt
done

fsl_sub -N bet_jobs -t commands.txt

For more details on how to use the fsl_sub options see the Advanced Usage section.

Warning

The command you submit cannot run any graphical interface, as they will have no where to display the output.

Jobs requiring GPUs

Whilst GPU tasks can simply be submitted to the gpu_short.q or gpu_long.q paritions fsl_sub also provides helper options which can automatically select a GPU queue and select the appropriate CUDA toolkit for you.

  • ​-c|--coprocessor <coprocessor name>: This selects the coprocessor with the given name (see fsl_sub --help for details of available coprocessors)
  • --coprocessor_multi <number>: This allows you to request multiple GPUs. On the FMRIB cluster you can select no more than two GPUs. You will automatically be given a two-slot openmp parallel environment
  • --coprocessor_class <class>: This would allow you to select which GPU hardware model you require, see fsl_sub --help for details
  • --coprocessor_toolkit <toolkit version>: This allows you to select the API toolkit your sofware needs. This will automatically make available the requested CUDA libraries where these haven't been compiled into the software

There are several CUDA coprocessor definitions configured for fsl_sub:

  • cuda selects GPUs capable of high-performance double-precision workloads and would normally be used for queued tasks such as Eddy and BedpostX.
  • cuda_all selects all GPUs.
  • cuda_ml selects GPUs more suited to machine learning tasks, they typically have very poor double-precision performance, instead being optimised for single, half and quarter precision workloads - use these for tasks involving ML inference and development, although training may still be more optimal on the general purpose GPUs depending on the task this involves, ask the developer of the software for advice on this. In the case of the FMRIB SLURM cluster there is no difference in double precision capability for all our GPUs, it is only included to allow for straightforward porting of your scripts to BMRC's cluster.

GPU and queue aware tools will automatically request a GPU if they detect it.

There are six A30 units, each divided into two, providing for twelve 12GB GPU cards and two H100 units, one 80GB card and one divided into seven 10GB GPUs.

When indicating RAM requirements with -R you should consider the following quotation from the BMRC pages:​

Quote

When submitting jobs, the total memory requirement for your job should be equal to the the compute memory + GPU memory i.e. you will need to request a sufficient number of slots to cover this total memory requirement.

Interactive ​jobs (including gpu/machine learning tasks)

Where your program requires interaction you can select a GPU when requesting a VDI, graphical MATLAB, Jupyter or RStudio session.

Alternatively, within a VDI session, you can request a text only interactive session using:

salloc -p gpu_short --gres=gpu:1 --cpus-per-gpu=2 --mem-per-cpu=8G

(...wait for job allocation...)

srun --pty /bin/bash -l

There may be a delay running the salloc command runs while the system finds a suitable host. Adapt the options as required, the example above requests:

  • -p gpu_short - gpu_short partition (1.25 days)
  • --gres=gpu:1 - requests a single gpu, for a specific type use gpu:k40:1 and change the number to 2 to request two GPUs
  • --cpus-per-gpu=2 - requests two CPU cores for each GPU allocated.
  • --mem-per-cpu=8G - allocates 16GB of memory for the task.

The srun command then launches a terminal into this interactive job.

When you have finished, use the command exit twice to return to your original terminal.

How to monitor your jobs on the cluster

You may want to know how your jobs are doing, if they have finished running, or if there has been an error. The command for this is squeue:

squeue -u <username>

This will give you a table with information on your jobs. An example of which is shown below:

![Job status output](job_monitorying.png)
Job status output

Your jobs will produce two text files that are useful for troubleshooting, and output file and an error file:

  • jobName.oXXXXXXX : Output messages from the programme that you have submitted.
  • jobName.eXXXXXXX : Error messages logged by the programme that you have submitted. If no errors, this should be an empty file.

You can peek inside these files from the command line, e.g.:

cat jobName.eXXXXXXX

XXXXXXX refers to the 7-digit job ID.

Auto-submitting software

Some FSL commands and GUIs automatically queue themselves where appropriate, i.e., you do not need to use fsl_sub to submit these programs. Examples include FEAT, TBSS, and BedpostX. If you do submit one of these tools to the queues then they will still run, but may not be able to make full use of the cluster resources (e.g. not be able to run multiple tasks in parallel).

For these auto-submitting software, you can still specify the memory you expect the task to require with the environment variable FSLSUB_MEMORY_REQUIRED. For example:

FSLSUB_MEMORY_REQUIRED=32G feat mydesign.feat

will submit a FEAT task informing fsl_sub that you expect to require 32GiB of memory.