FSL conda packages

This page describes: - how to write a conda recipe for a FSL project - how to build a conda package from a FSL conda recipe

All FSL project repositories have an associated recipe repository, which contains metadata describing how to build a conda recipe from the project repository.

When creating a conda recipe for a new FSL project, the easiest way to start is to copy the recipe for an existing similar project. For example, if you are creating a recipe for a Python project, you could copy the eddy_qc recipe. Or, if you are creating a recipe for a C++ project, you could copy the avwutils recipe.

Conda recipes for all FSL projects, and some internally packaged/managed third-party dependencies, are hosted in the GitLab fsl/conda namespace. The name of a FSL conda recipe repository is usually the same as the name of the FSL conda package - for example, the conda package for the fsl/avwutils project is named fsl-avwutils, and is hosted at fsl/conda/fsl-avwutils.

FSL conda package naming conventions

FSL conda package names must follow the conda package naming conventions, and be comprised solely of lowercase alpha characters, numeric digits, underscores, hyphens, or dots.

Furthermore, all FSL conda packages are prefixed with fsl-. An FSL project with name <project> will have a corresponding conda package name of fsl-<project>. For FSL projects with a name that begins with fsl (e.g. fslvbm, fsl_deface), the leading fsl will be dropped in the construction of the corresponding conda-package name. For example:

FSL project name	Conda package name
avwutils	fsl-avwutils
fslvbm	fsl-vbm
fsl_deface	fsl-deface
fsl-mrs	fsl-mrs
NewNifti	fsl-newnifti

There are a small number of exceptions to the above conventions. For example, the fdt project is built into two conda packages - the fsl-fdt package, providing CPU-only executables, and the fsl-fdt-cuda package, providing GPU/CUDA-enabled executables.

Conda recipes for FSL CUDA projects

Most FSL CUDA projects usually provide both GPU-enabled and CPU-only executables. For example, the fsl/fdt provides a range of CPU-only executables, including dtifit and vecreg, in addition to providing GPU-enabled executables such as xfibres_gpu.

To accommodate this convention, multiple conda recipes are used for these "hybrid" projects. For example, packages for the fsl/fdt project are built from two separate recipes:

• fsl/conda/fsl-fdt, which builds the CPU-only executables - these recipes are built as linux and macos packages.
• fsl/conda/fsl-fdt-cuda, which builds the GPU-enabled executables - these recipes are built as linux-cuda-X.Y packages.

Creating a conda recipe for a FSL project

We recommend copying an existing FSL conda recipe to generate an initial version of the recipe for your FSL project. This has the advantage that recipes for all FSL projects will follow a few simple conventions and standards.

However, it is possible to create a conda recipe by hand.

The purpose of this section is not to provide full instructions on writing a conda recipe, but rather on the specific issues that need to be considered when writing a conda recipe for a FSL project. More general information on creating conda recipes can be found at these websites:

• https://docs.conda.io/projects/conda-build/en/latest/resources/define-metadata.html
• https://python-packaging-tutorial.readthedocs.io/en/latest/conda.html

A FSL conda recipe is simply a flat directory containing the following files:

• meta.yaml: A YAML file which contains a description of the conda package, including its name, current version, URL to the project repository, and list of dependencies.

• build.sh: A bash script which is executed in order to build the package. For FSL C++ projects, this script essentially just calls ${FSLDIR}/etc/fslconf/fsl-devel.sh, then runs make and make install. For FSL Makefile-based projects, build.sh is required, but for Python-based projects, build.sh may or may not be necessary.

• post-link.sh: For projects which provide executables that are installed into ${FSLDIR}/bin/, this script is used to conditionally create entry points/wrapper scripts in ${FSLDIR}/share/fsl/bin/, as outlined in the configuration page. This script is not required for projects which do not provide any executables.

• pre-unlink.sh: This script is called when a package is uninstalled - its job is to remove any entry points that were created by post-link.sh. This script is not required for projects which do not provide any executables.

FSL projects are broadly divided into one of the following categories:

• Makefile-based project: FSL projects which use a FSL-style Makefile to compile and install their deliverables (shared libraries, scripts, compiled executables, and/or data files).

• Makefile-based CUDA project: FSL projects which use a FSL-style Makefile, and which provide GPU-accelerated executables linked against CUDA.

• Python project: FSL projects which are written in Python, and which have a setup.py or pyproject.toml file which is used to build the project as a Python package.

The mechanisms by which projects in these categories are built are slightly different and, therefore, the conda recipe for projects from different category will look slightly different. Some important details are highlighted below.

Writing the meta.yaml file

The meta.yaml file contains metadata about your project, including:

• The conda package name
• The URL of the project git repository
• The version number
• The build number (used in case the conda package for a single version needs to be re-built for some reason).
• A list of the package build- and run-time dependencies.

Essential metadata

In order to allow for automatic maintenance of FSL conda recipes, you should define the essential metadata (name, version, etc) as jinja2 variables using the following syntax:

{% set name       = '<conda-package-name>' %}
{% set version    = '<version-number>'     %}
{% set repository = '<repository-url>'     %}
{% set build      = '0'                    %}

Then use these variables within the recipe YAML, e.g.:

package:
  name:    {{ name    }}
  version: {{ version }}

By following this convention, FSL conda recipes can be automatically updated when a new version of a project is released.

Project repository and git revision

The automated CI rules defined in fsl-ci-rules allow the project repositoy and git revision (tag, branch, etc) used to build a conda package to be overridden via the FSLCONDA_REPOSITORY and FSLCONDA_REVISION environment variables. To facilitate this, the project source repository and git revision must be specified in the meta.yaml like so (remembering that we have defined repository and version as jinja2 variables, above):

source:
  # the FSLCONDA_REPOSITORY and FSLCONDA_REVISION
  # environment variables can be used to override
  # the repository/revision for development purposes.
  git_url: {{ os.environ.get("FSLCONDA_REPOSITORY", repository) }}
  git_rev: {{ os.environ.get("FSLCONDA_REVISION",   version)    }}

Following this convention allows conda packages for development and testing to be built, both automatically, and when developing/testing locally, simply by setting the FSLCONDA_REPOSITORY and FSLCONDA_REVISION variables.

run_exports (C/C++ projects only)

When compiling a C/C++ project, any shared library dependencies of the project must be present at the time of compilation, and at run time. This means that the dependencies of your project may need to be listed twice within the requirements section - once under host (or build), and again under run. We can avoid having to list dependencies twice by specifying run_exports in the dependency recipes.

The run_exports section is a trick which can be used within meta.yaml, which essentially allows us to define a dependency as a build-time dependency, and have it automatically propagated as a run-time dependency.

run_exports also allows us to specify the ABI compatibilty guarantees of a project. Maintainers of a C/C++ library must consider ABI compatibility across different versions of the library. For FSL C/C++ projects which follow the YYMM.B versioning scheme (described in the FSL project management page), different releases within a single YYMM series must preserve ABI compatibility.

The ABI compatibility guarantee that a particular project promises can be encoded in the run_exports section of the project conda recipe, by using the pin_subpackages(name, max_pin) macro function. As an example, for a project A which follows the major.minor.patch versioning scheme, setting max_pin to 'X.X' specifies that different releases of project A with the same major.minor version are ABI-compatible. For another package B which is dependent on A, conda will ensure that, when B is installed, an ABI-compatible version of A will be installed alongside B.

For FSL projects which follow the YYMM.B versioning scheme, max_pin should be set to 'X', which indicates that different releases with the same YYMM prefix are ABI-compatible.

So if you are writing a conda recipe for a C/C++ project which provides shared library files that will be used by other projects, add a run_exports section to the build section, and set max_pin according to the versioning scheme used for the project, like so:

build:
  number: {{ build }}
  run_exports:
    strong:
      - {{ pin_subpackage(name, max_pin='X') }}

noarch and script (Python based projects only)

Conda recipes for Python projects are built slightly differently to native projects, as the Python package build machinery is integrated into the conda build process. For Python projects, the build command is usually specified within the build section of the meta.yaml file. Furthermore, if you are packging a pure Python project, with no natively compiled code or extensions, you must label your recipe as being of type noarch: python - this is also specified within the build section. So a typical build section for a Python project will resemble the following:

build:
  number: {{ build }}
  noarch: python
  script: {{ PYTHON }} -m pip install . --no-deps --ignore-installed --no-cache-dir -vvv

Requirements

The fsl/base project provides the fundamental elements of a FSL installation, including FSL initialisation scripts and the Makefile machinery. As such it must be installed in order to build Makefile-based FSL projects, and must be present at run time for most FSL commands to function. All FSL conda recipes must therefore list fsl-base as a host requirement, which will cause it to be automatically added as both a build and run requirement¹.

C/C++ projects will also need to have a C++ compiler installed at build time. For example:

requirements:
  build:
    - {{ compiler('cxx') }}
    - make
  host:
    - fsl-base

¹ This is because the fsl-base recipe uses the run_exports trick described above.

Recipes for CUDA projects

Separate packages may be created from Makefile-based CUDA projects for each supported CUDA version. For example, multiple packages are built from the fsl-fdt-cuda recipe - each package is called fsl-fdt-cudaX.Y, where X.Y denotes the CUDA version that the package was built against.

To enable this, the meta.yaml file for a CUDA project needs to contain a couple of additional elements. First, when a CUDA package is built, a variable called CUDA_VER, containing the major.minor CUDA version the package is being built against, must be set in the environment. The meta.yaml file can read this environment variable in as a jinja2 variable like so:

{% set cuda_version = os.environ["CUDA_VER"] %}

Built CUDA packages must be named according to the version of CUDA they were built against, with a suffix of the form -cuda-X.Y . This is accomplished by adding the CUDA version to the package name like so:

{% set cuda_label   = '-cuda-' + cuda_version %}
{% set name         = 'fsl-fdt' + cuda_label %}

When a CUDA package is built, the CUDA toolkit and nvcc compiler must be installed in the build environment independently of conda, but the C++ compiler should be installed via conda. One complication which needs to be addressed is that different versions of nvcc are compatible with different versions of gcc, so the version of gcc that should be installed depends on the version of CUDA against which the package is being built.

To handle this complication, the requirements section for a CUDA project needs to look something like this:

requirements:
  host:
    - fsl-base
  build:
    # Different versions of nvcc need
    # different versions of gcc. This is
    # outlined in the "System requirements"
    # section of the CUDA installation
    # guide for each CUDA version, e.g.:
    # https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html
    {% if   cuda_version in ("9.2", "10.0") %}
    - {{ compiler("cxx") }} 7.*  # [linux]
    {% elif cuda_version in ("10.1", "10.2") %}
    - {{ compiler("cxx") }} 8.*  # [linux]
    {% elif cuda_version in ("11.0", "11.1", "11.2", "11.3") %}
    - {{ compiler("cxx") }} 9.*  # [linux]
    {% elif cuda_version in ("11.4", "11.5", "11.6", "11.7", "11.8") %}
    - {{ compiler("cxx") }} 11.*  # [linux]
    {% elif cuda_version in ("12.0", "12.1", "12.2", "12.3") %}
    - {{ compiler("cxx") }} 12.*  # [linux]
    {% elif cuda_version in ("12.4", "12.5") %}
    - {{ compiler("cxx") }} 13.*  # [linux]
    {% else %}
    - {{ compiler("cxx") }} # [linux]
    {% endif %}
    - make

When conda build calls the recipe build.sh script, it sanitises the environment variables that are passed according to a set of white-listed variables. In order for the build process to find the nvcc compiler executable, you must ensure that either: - nvcc is available via the ${PATH} environment variable when conda build is called, or - The ${NVCC} environment variable points to nvcc, and the recipe meta.yaml contains: build: # The FSL build system expects NVCC to # contain the path to the nvcc compiler script_env: - NVCC

Defining the build proceess

If you are writing a recipe for a Makefile-based FSL project, you need to write a build.sh script which uses the FSL Makefile machinery to build your project. A typical build.sh script will resemble the following:

#!/usr/bin/env bash

# $PREFIX is the installation destination
# when a conda package is being built
export FSLDIR=${PREFIX}

# Configure the build environment
. ${FSLDIR}/etc/fslconf/fsl-devel.sh

# Inject FSL copyright boilerplate
# into project source code
make insertcopyright

# Install project source code into
# ${PREFIX}/src/ (a.k.a. ${FSLDIR}/src/)
mkdir -p ${PREFIX}/src/
cp -r $(pwd) ${PREFIX}/src/${PKG_NAME}

# Build and install the project
make
make install

If you are writing a recipe for a Python-based FSL project, a build.sh script is generally not required.

Notes on CUDA projects

A critical requirement of "hybrid" FSL CUDA projects, which provide both CPU-only and GPU-capable executables, is that the project Makefile must be able to conditionally compile only the CPU components, or the GPU components, provided by the project.

The specific make invocations that need to be made depend on how the project Makefile is written. For example, the fdt Makefile accepts cpu and gpu flags, to control which parts of the project are compiled - make cpu=1 will compile only the CPU components of the fdt project, whereas make cpu=0 gpu=1 will compile only the GPU components.

The build.sh scripts for the fsl-fdt and fsl-fdt-cuda take advantage of this mechanism, so that the fsl-fdt recipe will only compile the CPU components, and the fsl-fdt-cuda recipe will only compile the GPU components

The fdt Makefile generates labelled binaries - CUDA executables and shared libraries are named according to the version of CUDA they were compiled against, e.g. xfibres_gpu9.2. This is important so that different variants of the xfibres_gpu executable, compiled against different CUDA versions, can be installed alongside each other in the same environment.

The FSL Makefile-based build system allows CUDA binaries to either statically or dynamically link against the CUDA Toolkit. Static linking is the default behaviour - this allows binaries to be used without requiring the CUDA Toolkit to be installed.

However, dynamic linking may be preferable for local development - it can be enabled by passing the CUDA_DYNAMIC=1 variable to the make command.

The build.sh script for the fsl-fdt recipe therefore looks the same as the template build.sh script above, except the make invocations are as follows:

make cpu=1
make cpu=1 install

And the make invocations in the build.sh script for the fsl-fdt-cuda recipe are as follows:

make cpu=0 gpu=1
make cpu=0 gpu=1 install

post-link.sh and pre-unlink.sh scripts.

Official/full FSL installations contain entry points/wrapper scripts for every FSL executable in ${FSLDIR}/share/fsl/bin/ - this is so that a user can add FSL commands to their ${PATH} via this directory, rather than the ${FSLDIR}/bin/ directory, and avoid adding all of the other executables in ${FSLDIR}/bin/ to their ${PATH} (e.g. python).

These wrapper scripts are created/removed by two utility commands which are installed by the fslinstaller.py script - createFSLWrapper and removeFSLWrapper. The conda recipes for any FSL projects which provide executables need to call these scripts at the time of installation/uninstallation to ensure that wrapper scripts for the executables are created/removed.

This can be achieved by using post-link.sh and pre-unlink.sh scripts, which are conda-specific mechanisms allowing custom logic to be executed when a conda package is installed or uninstalled.

For a FSL project which provides executables called fsl_command1, fsl_command2 and fsl_command3, the post-link.sh for the project recipe should contain:

if [ -e ${FSLDIR}/share/fsl/sbin/createFSLWrapper ]; then
    ${FSLDIR}/share/fsl/sbin/createFSLWrapper fsl_command1 fsl_command2 fsl_command3
fi

The corresponding pre-unlink.sh script should contain:

if [ -e ${FSLDIR}/share/fsl/sbin/removeFSLWrapper ]; then
    ${FSLDIR}/share/fsl/sbin/removeFSLWrapper fsl_command1 fsl_command2 fsl_command3
fi

Building a FSL conda package locally

This section describes how to build a conda package for an FSL project from the corresponding conda recipe.

Normally there should be no need to build FSL conda packages by hand, as packages are automatically built and published using Gitlab CI. However the need may arise for development, testing, or debugging purposes, and hence the process is described here.

In order to build a FSL conda package locally, all you need is a (mini)conda environment with conda-build installed. You do not need to have a compiler, or even FSL, installed. If you are working with internal-only FSL conda packages, you may also need a username and password to access the internal FSL conda channel.

Note: If you are building a CUDA package, you need to have a CUDA Toolkit installed, and the nvcc executable available on your ${PATH}.

Step 1: Clone the recipe repository

To start, you need to create a local clone of the recipe repository. For example, if you would like to build a package for fsl/avwutils:

git clone https://git.fmrib.ox.ac.uk/fsl/conda/fsl-avwuitls

Step 2: Clone the project repository (optional)

By default, the fsl-avwutils recipe will build a package from the fsl/avwutils gitlab repository. If you would like to build a package from a local clone, or a personal fork, of fsl/avwutils, you can use the FSLCONDA_REPOSITORY variable to override the default setting (which is to build from the gitlab fsl/avwutils project):

git clone https://git.fmrib.ox.ac.uk/fsl/avwuitls.git
export FSLCONDA_REPOSITORY=$(pwd)/avwutils

Note: It is not currently possible to build a conda package from a local copy of a project which is not a git repository, nor from a dirty working tree (i.e. the changes you want to build must be committed). This functionality may be added in the future if it is deemed necessary.

Step 3: Choose the revision you want to build (optional)

By default, the fsl-avwutils conda recipe will build a package from the last released version (tag) on the fsl/avwutils gitlab repository - this is specified in in the recipe meta.yaml. If you would like to build a package for a different release, or for a specific branch, you can set the FSLCONDA_REVISION variable to override the default setting. For example, if you want to build a package for the main branch:

export FSLCONDA_REVISION=main

Step 4: Run conda build to build the package

Now you can run conda build to build the conda package. The channel order is very important, and must be:

1. The public FSL conda channel has the highest priority.
2. conda-forge Packages from conda-forge have lower priority.

cd fsl-avwutils
conda build                                                            \
  -c https://fsl.fmrib.ox.ac.uk/fsldownloads/fslconda/releases/public/ \
  -c conda-forge                                                       \
  --output-folder=dist                                                 \
  .

If your project depends on internal-only FSL packages, add the internal FSL conda channel before the public channel, like so (set the ${FSLCONDA_USERNAME} and ${FSLCONDA_PASSWORD} environment variables to the username/password for the internal FSL conda channel):

cd fsl-avwutils
conda build                                                                                               \
  -c https://${FSLCONDA_USERNAME}:${FSLCONDA_PASSWORD}@fsl.fmrib.ox.ac.uk/fsldownloads/fslconda/internal/ \
  -c https://fsl.fmrib.ox.ac.uk/fsldownloads/fslconda/public/                                             \
  -c conda-forge                                                                                          \
  --output-folder=dist                                                                                    \
  .

Similarly, if you wish to build your package using development versions of other FSL conda packages, specify the FSL development conda channel before the public channel, e.g.:

conda build                                                        \
  -c https://fsl.fmrib.ox.ac.uk/fsldownloads/fslconda/development/ \
  -c https://fsl.fmrib.ox.ac.uk/fsldownloads/fslconda/public/      \
  -c conda-forge                                                   \
  --output-folder=dist                                             \
  .

If the build succeeds, the built package will be saved to the dist directory - binary packages will be saved in either the osx-64 or linux-64 sub-directories, and platform-independent (e.g. python) packages will be saved in the noarch sub-directory.

Building multiple FSL conda packages locally

The need may arise to build conda packages for multiple FSL projects at once. For example, you might be making changes to one project which depends on changes you have made to another project (a dependency of the first project). In this case it makes sense to set up a local conda channel, and to build/install the dependencies into that channel.

Let's say we are working on the randomise project, and are simultaneously making changes to the newimage project. We will need local clones of all the project and recipe repositories:

git clone https://git.fmrib.ox.ac.uk/fsl/randomise.git
git clone https://git.fmrib.ox.ac.uk/fsl/newimage.git
git clone https://git.fmrib.ox.ac.uk/fsl/conda/fsl-randomise.git
git clone https://git.fmrib.ox.ac.uk/fsl/conda/fsl-newimage.git

Before we can build a conda package for randomise, we need to build a conda package from our local development version of newimage. We use FSLCONDA_REPOSITORY and FSLCONDA_REVISION to direct the build to use our local newimage repository, and direct the build to my_local_conda_channel, which we will use later.

mkdir my_local_conda_channel

export FSLCONDA_REPOSITORY=$(pwd)/newimage
export FSLCONDA_REVISION=enh/my_local_newimage_development_branch

conda build                                                   \
  -c https://fsl.fmrib.ox.ac.uk/fsldownloads/fslconda/public/ \
  -c conda-forge                                              \
  --output-folder=./my_local_conda_channel                    \
  ./fsl-newimage

Now we can build our development version of randomise, using the development conda package we just built for newimage, simply by adding our local channel directory as a conda channel, making sure to list it before the FMRIB conda channel URL, so that it takes precedence:

export FSLCONDA_REPOSITORY=$(pwd)/randomise
export FSLCONDA_REVISION=enh/my_local_randomimse_development_branch

conda build                                                   \
  -c file://$(pwd)/my_local_conda_channel                     \
  -c https://fsl.fmrib.ox.ac.uk/fsldownloads/fslconda/public/ \
  -c conda-forge                                              \
  --output-folder=./my_local_conda_channel                    \
  ./fsl-randomise