Skip to content

Using containers for python environments

Introduction

Using containers for Python environments can greatly simplify dependency management and ensure reproducibility. Containers encapsulate all dependencies, libraries, and configurations needed to run your Python code, making it easier to share and deploy across different systems.

In this guide, we will explore how to create and manage Python environments using containers, specifically focusing on tools like Apptainer / Singularity.

Getting Started

To get started with containers for Python environments, you will need to have a containerization tool installed. Apptainer (formerly Singularity) is a popular choice for scientific computing environments. (You can install Apptainer on your system by following the official installation guide: Apptainer Installation) On IMGW servers, VSC and ECMWF this is already installed. If the runtime is not directly available, you can use the module load apptainer command to load it.

Once the container is build, you can run it with the following command:

Bash
1
2
3
4
5
6
7
8
apptainer run my_container.sif
# or
./my_container.sif

# or running a command inside the container
apptainer exec my_container.sif <command>
# or
./my_container.sif <command>
The container will automatically use the local directory and bind it to the container, so you can access your files inside the container. If you need to access non-standard directories, you can set the SINGULARITY_BIND environment variable to include those directories. For example:

Bash
1
2
3
4
5
6
# using the container with access to /srvfs, /jetfs, and /gpfs
export SINGULARITY_BIND="/srvfs,/jetfs,/gpfs"
./my_container.sif python3 my_script.py /jetfs/my_data/input.txt

# or using the apptainer command directly
apptainer run --bind /srvfs,/jetfs,/gpfs my_container.sif python3 my_script.py /jetfs/my_data/input.txt

Example: Creating a Python Environment with Apptainer

There is a tool micromamba2container.sh available here that can help you create a container with a specific Python environment. This script uses Micromamba to create a lightweight container with the desired Python version and packages.

Bash
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
./micromamba2container.sh -h
Usage: ./micromamba2container.sh -c <containername> -n <envname> -f <environment_file> -v <version>
Options:
    -c <containername>   Name of the container (default: micromamba)
    -n <envname>         Name of the micromamba environment to export (default: base)
    -f <environment_file>  Path to a micromamba environment YAML file (optional)
    -v <version>         Version of the container, only for naming (optional, e.g. 20240601 or latest)
    -h                   Show this help message

This script requires micromamba and apptainer (1.1.0+) to be installed.

# Create a simple environment with some dependencies
./micromamba2container.sh -c my_container 
Using micromamba to create a new environment...
Enter Python version (default is 3.12): 3.13
If you mistyped, you can edit the Apptainer.recipe file later.
Enter all packages to install (space-separated): eccodes earthkit-data netcdf4 xarray matplotlib cartopy pandas numpy scipy dask
Creating Apptainer recipe for container 'my_container' with environment ''...
You can edit that file manually if you want to add more packages or change the base image.
Rebuild: apptainer build --ignore-fakeroot-command --ignore-subuid my_container.sif Apptainer.recipe

------------------------------------------------
Building Apptainer container...
INFO:    User not listed in /etc/subuid, trying root-mapped namespace
INFO:    fakeroot command not found
INFO:    Installing some packages may fail
WARNING: 'nodev' mount option set on /tmp, it could be a source of failure during build process
INFO:    Starting build...
INFO:    Fetching OCI image...
...
Cleaning tarballs..
Cleaning packages..
INFO:    Adding labels
INFO:    Adding environment to container
INFO:    Adding runscript
INFO:    Creating SIF file...
INFO:    Build complete: my_container.sif
Apptainer container 'my_container.sif' created successfully from environment ''.
------------------------------------------------
Access non standard directories by setting the SINGULARITY_BIND environment variable.

For example, to include /srvfs, /jetfs or /gpfs (VSC5), run:
export SINGULARITY_BIND="/srvfs,/jetfs,/gpfs"

You can now run the container with:
./my_container.sif
apptainer run my_container.sif
apptainer shell my_container.sif

To execute a command in the container, use:
./my_container.sif <command>
apptainer exec my_container.sif <command>

------------------------------------------------
Remember that the micromamba environment is located in /opt/conda.
You can also edit the Apptainer.recipe file to add more packages or change the base image.
To rebuild the container, run:
apptainer build --ignore-fakeroot-command --ignore-subuid my_container.sif Apptainer.recipe

------------------------------------------------

# Now you can run this container:
./my_container.sif
No arguments provided. Running default shell.
[my_container]~/home$ python3
[my_container]~/home$ exit

Modify the recipe file to add pip packages

There are some limitations, when you do it like this. You can not add pip packages like this. You can modify the recipe file to add pip packages. e.g.

Text Only
1
2
3
%post
    micromamba install -y -n base -c conda-forge python=3.13 eccodes earthkit-data netcdf4 xarray matplotlib cartopy pandas numpy scipy dask
    pip install --no-cache-dir some-pip-package

or use an environment.yml with a pip section like this:

YAML
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
name: myenv
channels:
  - conda-forge
dependencies:
  - python=3.13
  - eccodes
  - earthkit-data
  - netcdf4
  - xarray
  - matplotlib
  - cartopy
  - pandas
  - numpy
  - scipy
  - dask
  - pip:
    - earthkit-plots
    - earthkit-meteo

Creating from an existing environment

You can create a new container from an existing micromamba environment by specifying the environment name with the -n option:

Bash
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# Check your environements
micromamba env list
  Name          Active  Path                                        
──────────────────────────────────────────────────────────────────────
  base          *       /home/user/micromamba                  
  ansible               /home/user/micromamba/envs/ansible     
  jupyterhub            /home/user/micromamba/envs/jupyterhub  
  myenv                 /home/user/micromamba/envs/myenv

# Create the container for the myenv environment
./micromamba2container.sh -c my_container -n myenv

This will create a new container 'my_container' from the existing micromamba environment 'myenv'. Existing environments are easy, because micromamba already has all the information about the packages and dependencies in that environment, so it can directly export them into the container.

Integration into Jupyterhub

In order to use the container in Jupyterhub, you can specify the path to the container in the jupyter configuration. the only dependency is that the container has the ipykernel package installed.

Bash
1
2
3
4
5
# Add the container to Jupyterhub
# Create the configuration file for the container
./my_container.sif python3 -m ipykernel install --user --name MYENV --display-name "My(3.13)"
Running with arguments: python3 -m ipykernel install --user --name MYENV --display-name My(3.13)
Installed kernelspec MYENV in /home/user/.local/share/jupyter/kernels/myenv

This is great, but of course the runtime needs to be added. So open that file:

Bash
1
vim ~/.local/share/jupyter/kernels/myenv/kernel.json

and add the lines marked with + to the file, so it looks like this:

JSON
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
{
 "argv": [
+    "apptainer",
+    "exec",
+    "--cleanenv",
+    "--bind",
+    "/srvfs,/jetfs,/gpfs",
+    "/path/to/my_container.sif",
    "/opt/conda/bin/python",
    "-m",
    "ipykernel_launcher",
    "-f",
    "{connection_file}"
 ],
 "display_name": "My(3.13)",
 "language": "python"
}

This ensures, that the container is launched with the correct environment and that the necessary directories are bound to the container. Make sure to use the absolute path for your container file. You can now select the kernel "My(3.13)" in Jupyterhub and it will run your code inside the container with the specified Python environment.