How to pass environment variables to SLURM jobs
By default no environment variables from your current shell are passed to your job running on the cluster.
Where the important variables were set by loading an environment module, you do not need to do anything as fsl_sub will automatically load the currently loaded modules in your cluster job, but for other variables you can request that fsl_sub pass a sub-set of variables to your job with the --export option (pass this multiple times to export multiple variables). You can also use this option to set an environment variable in your job that are not already set in your shell.
For very complicated use cases or dynamic variable setting, create a script that sets up all your variables and then calls the software - submit this script to the cluster.
There are two ways to use --export:
--export VARIABLENAME {--export VARIABLENAME}- This will copy the current environment variable setting into your job (specify multiple times for multiple variables)--export VARIABLENAME=VARIABLEVALUE- This will set the environment variable to the value after the '=' in the queued job only (not effecting your shell) so is ideal where you need to specify a job specific value
fsl_sub will automatically transfer some internal variables and may have been configured to include some additional useful ones, see the 'exports:' option in the output of
for the list of default exports. Any --export passed on the command line will override these configured options. It is also possible to configure fsl_sub for your account to always copy over particular variables, see Configuring fsl_sub.
Option 2, where you provide the variable with a value is particularly useful if you are scheduling many similar tasks and need to specify a different value for an environment variable for each run, for example SUBJECTS_DIR which FreeSurfer uses to specify where your data sets reside.
Environment variables that can be set to control fsl_sub submitted tasks
fsl_sub sets or can be controlled with the following shell variables. These can be set either for the duration of the fsl_sub run by prepending the call with the setting of the value:
or by exporting the value to your shell so that all subsequent calls will also have this variable set this way:
| Envrionment variable | Who sets | Purpose | Example values |
|---|---|---|---|
| FSLSUB_JOBID_VAR | fsl_sub | Variable name of Grid job id | SLURM_JOB_ID |
| FSLSUB_ARRAYTASKID_VAR | fsl_sub | Variable name of Grid task id | SLURM_ARRAY_TASK_ID |
| FSLSUB_ARRAYSTARTID_VAR | fsl_sub | Variable name of Grid first task id | SLURM_ARRAY_TASK_MIN |
| FSLSUB_ARRAYENDID_VAR | fsl_sub | Variable name of Grid last task id | SLURM_ARRAY_TASK_MAX |
| FSLSUB_ARRAYSTEPSIZE_VAR | fsl_sub | Variable name of Grid step between task ids | SLURM_ARRAY_TASK_STEP |
| FSLSUB_ARRAYCOUNT_VAR | fsl_sub | Variable name of Grid number of tasks in array | SLURM_ARRAY_TASK_COUNT |
| FSLSUB_MEMORY_REQUIRED | You | Advise fsl_sub of expected memory required | 32G |
| FSLSUB_PROJECT | You | Name of Grid project to run jobs under | MyProject |
| FSLSUB_COPROCESSOR_PROJECT | You | Name of account to run GPU (--coprocessor) jobs on if automatic selection inappropriate |
gpu_MyProject |
| FSLSUB_COPROCESSOR_CLASS | You | Override automatic class selection, e.g. to limit to V-16 or better | V-16 (see fsl_sub --help for available classes) |
| FSLSUB_COPROCESSOR_CLASS_STRICT | You | Force a single class, i.e prevent running on more expensive hardware | 0 or 1 |
| FSLSUB_PARALLEL | You/fsl_sub | Control array task parallelism when running without a cluster engine (e.g. when a queued task itself submits an array task) | 4 (for four threads), 0 to let fsl_sub's shell plugin use all available cores |
| FSLSUB_CONF | You | Provides the path to the configuration file | /usr/local/etc/fslsub_conf.yml |
| FSLSUB_NSLOTS | fsl_sub | Variable name of Grid allocated slots | NSLOTS |
| FSLSUB_DEBUG | You/fsl_sub | Enable debugging in child fsl_sub | 1 |
| FSLSUB_PLUGINPATH | You | Where to find installed plugins (do not change this variable) | /path/to/folder |
| FSLSUB_NOTIMELIMIT | You | Disable notification of job time to the cluster | 1 |
Where a FSLSUB_* variable is a reference to another variable you need to read the content of the referred to variable, e.g., to discover which grid task-id this process has: