Apptainer on IT4Innovations¶
Singularity name change
On November 30th, 2021, the Singularity project has officially moved into the Linux Foundation. As part of this move, and to differentiate from the other like-named projects and commercial products, the project is renamed to "Apptainer".
On our clusters, the Apptainer images of main Linux distributions are prepared.
Barbora Karolina ├── CentOS ├── CentOS | └── 7 | └── 7 ├── Rocky ├── Rocky | ├── 8 | ├── 8 │ └── 9 │ └── 9 ├── Fedora ├── Fedora │ └── latest │ └── latest └── Ubuntu └── Ubuntu └── latest └── latest
Current information about available Apptainer images can be obtained by the
ml av command. The images are listed in the
The bootstrap scripts, wrappers, features, etc. are located on it4i-singularity GitLab page.
IT4Innovations Apptainer Wrappers¶
For better user experience with Apptainer containers, we prepared several wrappers:
Listed wrappers help you to use prepared Apptainer images loaded as modules.
You can easily load a Apptainer image like any other module on the cluster by the
ml OS/version command.
After the module is loaded for the first time, the prepared image is copied into your home folder and is ready for use.
When you load the module next time, the version of the image is checked and an image update (if exists) is offered.
Then you can update your copy of the image by the
With an image update, all user changes to the image will be overridden.
The runscript inside the Apptainer image can be run by the
CentOS/7 module only
This command automatically mounts the
/apps storage and invokes the image as writable, so user changes can be made.
Very similar to
image-run is the
The only difference is that
image-exec runs a user-defined command instead of a runscript.
In this case, the command to be run is specified as a parameter.
Using the interactive shell inside the Apptainer container is very useful for development.
In this interactive shell, you can make any changes to the image you want,
but be aware that you can not use the
sudo privileged commands directly on the cluster.
To simply invoke interactive shell, use the
Another useful feature of the Apptainer is the direct support of OpenMPI.
For proper MPI function, you have to install the same version of OpenMPI inside the image as you use on the cluster.
OpenMPI/4.1.2 is installed in prepared images (CentOS 7, Rocky 8).
The MPI must be started outside the container.
The easiest way to start the MPI is to use the
This command has the same parameters as
mpirun, so there is no difference between running normal MPI application
and MPI application in a Apptainer container.
In the examples, we will use prepared Apptainer images.
$ ml CentOS/7 Preparing image CentOS-7_20230116143612.sif 261.20M 100% 412.36MB/s 0:00:00 (xfr#1, to-chk=0/1) Your image of CentOS/7 is at location: /home/username/.apptainer/images/CentOS-7_20230116143612.sif
After the module is loaded for the first time, the prepared image is copied into your home folder to the .apptainer/images subfolder.
Executes the given command inside the Apptainer image. The container is in this case started, then the command is executed and the container is stopped.
$ ml CentOS/7 $ image-exec cat /etc/redhat-release CentOS Linux release 7.9.2009 (Core)
MPI wrapper - see more in the Examples MPI section.
This command runs the runscript inside the Apptainer image. Note, that the prepared images do not contain a runscript.
Invokes an interactive shell inside the Apptainer image.
$ ml CentOS/7 $ image-shell Apptainer>
This command is for updating your local Apptainer image copy. The local copy is overridden in this case.
$ ml CentOS/7 New version of CentOS image was found. (New: CentOS-7_20230116143612.sif Old: CentOS-7_20230115143612.sif) For updating image use: image-update Your image of CentOS/7 is at location: /home/username/.apptainer/images/CentOS-7_20230115143612.sif $ image-update New version of CentOS image was found. (New: CentOS-7_20230116143612.sif Old: CentOS-7_20230115143612.sif) Do you want to update local copy? (WARNING all user modification will be deleted) [y/N]: y Updating image CentOS-7_20230116143612.sif 2.71G 100% 199.49MB/s 0:00:12 (xfer#1, to-check=0/1) sent 2.71G bytes received 31 bytes 163.98M bytes/sec total size is 2.71G speedup is 1.00 New version is ready. (/home/username/.apptainer/images/CentOS-7_20230116143612.sif)
In the following example, we are using a job submitted by the command:
qsub -A PROJECT -q qprod -l select=2:mpiprocs=128 -l walltime=00:30:00 -I
We have seen no major performance impact for a job running in a Apptainer container.
With Apptainer, the MPI usage model is to call
mpirun from outside the container
and reference the container from your
Usage would look like this:
$ mpirun -np 128 apptainer exec container.img /path/to/contained_mpi_prog
mpirun outside of the container, we solve several very complicated work-flow aspects.
For example, if
mpirun is called from within the container, it must have a method for spawning processes on remote nodes.
Historically the SSH is used for this, which means that there must be an
sshd running within the container on the remote nodes
sshd process must not conflict with the
sshd running on that host.
It is also possible for the resource manager to launch the job
and (in OpenMPI’s case) the Orted (Open RTE User-Level Daemon) processes on the remote system,
but that then requires resource manager modification and container awareness.
In the end, we do not gain anything by calling
mpirun from within the container
except for increasing the complexity levels and possibly losing out on some added
performance benefits (e.g. if a container was not built with the proper OFED as the host).
MPI Inside Apptainer Image¶
$ ml CentOS/7 $ image-shell Apptainer> mpirun hostname | wc -l 128
As you can see in this example, we allocated two nodes, but MPI can use only one node (128 processes) when used inside the Apptainer image.
MPI Outside Apptainer Image¶
$ ml CentOS/7 Your image of CentOS/7 is at location: /home/username/.apptainer/images/CentOS-7_20230116143612.sif $ image-mpi hostname | wc -l 256
In this case, the MPI wrapper behaves like the
mpirun command is called outside the container
and the communication between nodes are propagated into the container automatically.
How to Use Own Image on Cluster?¶
- Prepare the image on your computer
- Transfer the images to your
/homedirectory on the cluster (for example
local:$ scp container.img firstname.lastname@example.org:~/.apptainer/image/container.img
- Load module Apptainer (
- Use your image
If you want to use the Apptainer wrappers with your own images, load the
apptainer-wrappers/1.0 module and set the environment variable
How to Edit IT4Innovations Image?¶
- Transfer the image to your computer
local:$ scp email@example.com:/home/username/.apptainer/image/container.img container.img
- Modify the image
- Transfer the image from your computer to your
/homedirectory on the cluster
local:$ scp container.img firstname.lastname@example.org:/home/username/.apptainer/image/container.img
- Load module Apptainer (
- Use your image