Skip to content

Python related Problems

Introduction to conda / micromamba

What is conda ?

This is the package manager for the anaconda or miniconda distribution of Python. It allows to install packages and create environments and much much more. However, mostly you will use it to install packages and create environments for your code. It is slow. link

What is micromamba ?

This is a replacement package manager for Python that is independent from a distribution and can be used to create fuly independent python environments with just the micromamba executable as strting point. link

Conda environment
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# load the module on department servers
module load micromamba
# Download and extract just micromamba exe
wget -qO- https://micromamba.snakepit.net/api/micromamba/linux-64/latest | tar -xvj --strip-components=1 bin/micromamba
# show
micromamba info
                                           __
          __  ______ ___  ____ _____ ___  / /_  ____ _
         / / / / __ `__ \/ __ `/ __ `__ \/ __ \/ __ `/
        / /_/ / / / / / / /_/ / / / / / / /_/ / /_/ /
       / .___/_/ /_/ /_/\__,_/_/ /_/ /_/_.___/\__,_/
      /_/


            environment : None (not found)
           env location : -
      user config files : /home/user/.mambarc
 populated config files :
       libmamba version : 1.1.0
     micromamba version : 1.1.0
           curl version : libcurl/7.87.0 OpenSSL/1.1.1s zlib/1.2.13 libssh2/1.10.0 nghttp2/1.47.0
     libarchive version : libarchive 3.6.2 zlib/1.2.13 bz2lib/1.0.8 libzstd/1.5.2
       virtual packages : __unix=0=0
                          __linux=6.1.12=0
                          __glibc=2.37=0
                          __archspec=1=x86_64
               channels :
       base environment : /home/user/micromamba
               platform : linux-64

What should I use?

It is recommended to use micromamba, which is way faster than conda. It has a smaller footprint and it works almost identical as conda. It is a c++ replacement of conda/mamba.

install packages

There are sometimes different versions available with different build options as well, you can use the search function (micromamba search) to get an understanding of what versions and builds are available.

micromamba environment
1
2
3
4
5
6
# load micromamba module
module load micromamba
# Create a new environment:
micromamba create -n <my-environment> <package_name>=<version>=<build_string>
# Install packages into an existing environment
micromamba install -c conda-forge -n <my-environment> <package_name>=<version>=<build_string>
Looking for Python packages/names?

Search here: conda-forge or PyPI

Creating a common environment for most users

It is more efficient to use a common environment, that is present on all servers and just add your few additions, then to spawn a full new copy of everything. It is very efficient to create an environment file and reuse these or build on top of an existing environment.

There are three environments present all department servers:

  1. Default Python environment present with a jupyterhub
  2. The Magic Python Kernel (definition)
  3. The RTTOV Python Kernel (container definition on Phaidra)
Create the Magic Kernel
1
2
3
4
5
6
# get the definition
$ wget https://wolke.img.univie.ac.at/documentation/general/Python/Magic.yml
# load micromamba
$ module load micromamba
# create the environment
$ micromamba create -n MagicPy -c conda-forge -f Magic.yml

Q: Installing Cartopy or any other module based environment?

The user needs to load geos and proj modules and then install the package via pip install --user cartopy or create an environment and install there.

Installing from conda-forge is also possible, without loading any modules. e.g. micromamba install -n <my-env> -c conda-forge cartopy in a conda environment. This will install a lot of files and libraries.

Go to: Cartopy Installation Notebook

Q: Load Modules into the current Notebook Environment?

The Idea is to create a load_modules_into_jupyter.conf file in your home directory asking for modules to be loaded when ther kernel is launched. Please note that the file needs an empty line at the end

Go to: Modules loaded in Kernel

Q: How to create a user environment?

The idea is to install a micromamba or virtual environment and use that as a kernel for your notebooks. Please note the solution for cartopy

Go to: Conda/Virtual Environment (updated 2025/06)

Q: The Notebook server on the TeachingHub won't start?

This usually happens because of an old configuration. Try to remove ~/.jupyter on SRVX1. Deprecated

Go to: Teaching-Hub Spawn

Q: Executing Python code on remote servers?

The idea is to launch an interactive IPython console on your computer (Linux or Mac, not sure about Windows) with a kernel on the server.

Go to: Remote IPython Kernels

Q: What Python IDE to use with remote kernels?

There are a lot of different ways to develop in python. Some are: - Visual Studio Code - PyCharm - Editors like Emacs, Sublime, Vim,... - Jupyter Notebooks and Lab e.g. on Jupyterhub@srvx1 or jupyterhub@jet

and atom. This is simple editor available on any platform and it can be extended with packages to become a full grown IDE with the power of open source.

Q: How to restore a conda environment?

Ever needed to backup a conda environment or copy from a colleague or designing one by hand to give to others?

Go to: Save/Restore conda environment

Q: How to get the dask dashboard working on Jupyterhub?

You just need to add a configuration option and then you can open the Dashboard in any Jupyterhub. Works on SRVX1 and JET.

Go to: Fix Dask Dashboard

Q: How to profile memory and exeution time of functions?

If you need to get a better understanding of you functions memory and execution time, try these profiling options.

Go to: Profile

Q: How to read BUFR files with python?

ECMWF created a python package that helps to read BUFR messages via ECCODES into pandas DataFrame.

Go to: BUFR

Q: How to read ODB files with python?

ECMWF created a python package that helps to read ODB messages via ODC or pure Python into pandas DataFrame.

Go to: ODB

Q: How to retrieve data from MARS in python?

ECMWF has created a package that allows to retrieve data from MARS, easily.

Go to: MARS

Q: How to retrieve ERA5 from CDS and plot in Magics?

It is very easy to download and plot ERA5 data, using cdsapi and Magics.

Go to: Magics

Q: How to ignore user site packages?

When using python it is possible to have multiple site, where packages can be installed. the default is to use the path from the python interpreter and a user site.

Bash
1
2
3
4
5
6
7
$ python -c 'import sys;print("\n".join(sys.path))'

/jetfs/manual/enstools/v2021.11/lib/python38.zip
/jetfs/manual/enstools/v2021.11/lib/python3.8
/jetfs/manual/enstools/v2021.11/lib/python3.8/lib-dynload
/jetfs/home/USER/.local/lib/python3.8/site-packages
/jetfs/manual/enstools/v2021.11/lib/python3.8/site-packages

this shows that there is a user site in your HOME directory for that python version. Python will always look for a directory for it's own version.

You can disable that by altering the sys.path (python type list) manually or using an environment variable (export PYTHONNOUSERSITE=1).

Bash
1
2
3
4
5
6
7
8
9
export PYTHONNOUSERSITE=1
$ python -c 'import sys;print("\n".join(sys.path))'

/jetfs/manual/enstools/v2021.11/lib/python38.zip
/jetfs/manual/enstools/v2021.11/lib/python3.8
/jetfs/manual/enstools/v2021.11/lib/python3.8/lib-dynload
/jetfs/manual/enstools/v2021.11/lib/python3.8/site-packages
# unset the variable again
unset PYTHONNOUSERSITE

Q: How to use pipelines for building and publishing packages?

Use a .gitlab-ci.yml file to automate building and publishing of your package.

Go to: Pipelines for building

Q: How do I migrate from Anaconda to venv or Micromamba?

There are easy alternatives, just use micromamba!

Go to: Migrate away from Anaconda

Q: How to convert a micromamba/mamba/conda environment into a container?

Why? You can convert you existing environment into just one file and run it even more efficient than having a hundreds of small files. Build your environment and take it with you. Every Server, HPC or Laptop can have apptainer installed and execute your kernel.

This can be an excellent way to deploy a conda environment on VSC.

Convert a conda environment into a container
Bash
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
# convert an existing conda environment into a container image
# get the script:
$ wget https://wolke.img.univie.ac.at/documentation/general/Python/micromamba2container.sh
# make executable
$ chmod +x micromamba2container.sh
# load micromamba and check the name of the environment
$ module load micromamba
$ micromamba env list
  Name     Active  Path
────────────────────────────────────────────────────────────────
  base     *       /jetfs/home/user/micromamba
  modern           /jetfs/home/user/micromamba/envs/modern
# ./micromamba2container.sh -c <containername> -n <envname> -v <version>
$ ./micromamba2container.sh -c PyModern -n modern
Using existing environment: modern
Creating Apptainer container from environment 'modern'...
Creating Apptainer recipe for container 'PyModern' with environment 'modern'...
You can edit that file manually if you want to add more packages or change the base image.
Rebuild: apptainer build --ignore-fakeroot-command --ignore-subuid PyModern.sif Apptainer.recipe
------------------------------------------------
Building Apptainer container...
INFO:    User not listed in /etc/subuid, trying root-mapped namespace
INFO:    fakeroot command not found
INFO:    Installing some packages may fail
INFO:    Starting build...
INFO:    Fetching OCI image...
26.9MiB / 26.9MiB [=========================================================] 100 % 2.6 MiB/s 0s
121.4KiB / 121.4KiB [=======================================================] 100 % 2.6 MiB/s 0s
6.7MiB / 6.7MiB [===========================================================] 100 % 2.6 MiB/s 0s
INFO:    Extracting OCI image...
INFO:    Inserting Apptainer configuration...
INFO:    Copying PyModern_env_20250701093559.yml to /opt/conda/PyModern_env_20250701093559.yml
INFO:    Running post scriptlet
+ micromamba -q install -y -n base --file /opt/conda/PyModern_env_20250701093559.yml
+ micromamba clean --all --yes
Collect information..
Cleaning index cache..
Cleaning lock files..
  Package file                                                  Size
──────────────────────────────────────────────────────────────────────
  /opt/conda/pkgs
──────────────────────────────────────────────────────────────────────
  _libgcc_mutex-0.1-conda_forge.tar.bz2                          3kB
  _openmp_mutex-4.5-2_gnu.tar.bz2                               24kB
  ...
──────────────────────────────────────────────────────────────────────
  Total size:                                                  240MB
Cleaning tarballs..
Cleaning packages..
INFO:    Adding labels
INFO:    Adding environment to container
INFO:    Adding runscript
INFO:    Creating SIF file...
[============================================================================] 100 % 0s
INFO:    Build complete: PyModern.sif
Apptainer container 'PyModern.sif' created successfully from environment 'modern'.
------------------------------------------------
Access non standard directories by setting the SINGULARITY_BIND environment variable.
For example, to include /srvfs, /jetfs or /gpfs (VSC5), run:
export SINGULARITY_BIND="/srvfs,/jetfs,/gpfs"
You can now run the container with:
./PyModern.sif
apptainer run PyModern.sif
apptainer shell PyModern.sif
To execute a command in the container, use:
./PyModern.sif <command>
apptainer exec PyModern.sif <command>
------------------------------------------------
Remember that the micromamba environment is located in /opt/conda.
You can also edit the Apptainer.recipe file to add more packages or change the base image.
To rebuild the container, run:
apptainer build --ignore-fakeroot-command --ignore-subuid PyModern.sif Apptainer.recipe
------------------------------------------------
# Done. Execute the container
$ ./PyModern.sif 
/PyModern.sif 
No arguments provided. Running default shell.
[PyModern]~$ ipython3
Python 3.12.0 | packaged by conda-forge | (main, Oct  3 2023, 08:43:22) [GCC 12.3.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.32.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import sys
In [2]: sys.path
Out[2]: 
['/opt/conda/bin',
'/opt/conda/lib/python312.zip',
'/opt/conda/lib/python3.12',
'/opt/conda/lib/python3.12/lib-dynload',
'',
'/opt/conda/lib/python3.12/site-packages']