Skip to content

Python related Problems

Introduction to conda / micromamba

What is conda ?

This is the package manager for the anaconda or miniconda distribution of Python. It allows to install packages and create environments and much much more. However, mostly you will use it to install packages and create environments for your code. It is slow. link

What is micromamba ?

This is a replacement package manager for Python that is independent from a distribution and can be used to create fuly independent python environments with just the micromamba executable as strting point. link

Conda environment
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# Download and extract just micromamba exe
wget -qO- https://micromamba.snakepit.net/api/micromamba/linux-64/latest | tar -xvj --strip-components=1 bin/micromamba
# show
micromamba info
                                           __
          __  ______ ___  ____ _____ ___  / /_  ____ _
         / / / / __ `__ \/ __ `/ __ `__ \/ __ \/ __ `/
        / /_/ / / / / / / /_/ / / / / / / /_/ / /_/ /
       / .___/_/ /_/ /_/\__,_/_/ /_/ /_/_.___/\__,_/
      /_/


            environment : None (not found)
           env location : -
      user config files : /home/user/.mambarc
 populated config files : 
       libmamba version : 1.1.0
     micromamba version : 1.1.0
           curl version : libcurl/7.87.0 OpenSSL/1.1.1s zlib/1.2.13 libssh2/1.10.0 nghttp2/1.47.0
     libarchive version : libarchive 3.6.2 zlib/1.2.13 bz2lib/1.0.8 libzstd/1.5.2
       virtual packages : __unix=0=0
                          __linux=6.1.12=0
                          __glibc=2.37=0
                          __archspec=1=x86_64
               channels : 
       base environment : /home/user/micromamba
               platform : linux-64

What should I use?

It is recommended to use micromamba, which is way faster than conda. It has a smaller footprint and it works almost identical as conda. It is a c++ replacement of conda/mamba.

install packages

There are sometimes different versions available with different build options as well.

Conda environment
1
2
3
4
5
6
# load the minimum module (alternative load anaconda3)
module load miniconda3
# Create an environment and install some packages
conda create --name <my-environment> <package_name>=<version>=<build_string>
# install packages into that environment again
conda install -c <channel> -n <my-environment> <package_name>=<version>=<build_string>
micromamba environment
1
2
3
4
5
# load micromamba exe (just one exe)
module load micromamba
#
micromamba create -n <my-environment> <package_name>=<version>=<build_string>
micromamba install -c conda-forge -n <my-environment> <package_name>=<version>=<build_string>

Q: Installing Cartopy on Jet Cluster or any other module based environment?

Cartopy Installation Notebook

The user needs to load geos and proj libraries and install via pip install --user cartopy or create an anaconda environment and install there.

Installing from conda-forge is also possible, without loading any modules. e.g. conda install -c conda-forge cartopy in a conda environment. This installs the libraries as well. (Not recommended)

Q: Load Modules into the current Notebook Environment?

Modules laoded in Kernel

The Idea is to create a load_modules_into_jupyter.conf file in your home directory asking for modules to be loaded when ther kernel is launched. Please note that the file needs an empty line at the end

Q: How to create a user environment?

Conda/Virtual Environment

The idea is to install a conda or virtual environment and use that as a kernel for your notebooks. Please note the solution for cartopy

Q: The Notebook server on the TeachingHub won't start?

Teaching-Hub Spawn

This usually happens because of an old configuration. Try to remove ~/.jupyter on SRVX1.

Q: Executing Python code on remote servers?

Remote IPython Kernels

The idea is to launch an interactive IPython console on your computer (Linux or Mac, not sure about Windows) with a kernel on the server.

Q: What Python IDE to use with remote kernels?

There are a lot of different ways to develop in python. Some are: - Visual Studio Code - PyCharm - Editors like Emacs, Sublime, Vim,... - Jupyter Notebooks and Lab e.g. on Jupyterhub@srvx1 or jupyterhub@jet

and atom. This is simple editor available on any platform and it can be extended with packages to become a full grown IDE with the power of open source.

Q: How to add a new package to anaconda module?

Add Package

It is relativly easy to install additional packages to a anaconda module distribution with pip install --user.

Q: How to restore a conda environment?

Save/Restore conda environment

Ever needed to backup a conda environment or copy from a colleague or designing one by hand to give to others?

Q: How to get the dask dashboard working on Jupyterhub?

Fix Dask Dashboard

You just need to add a configuration option and then you can open the Dashboard in any Jupyterhub. Works on SRVX1 and JET.

Q: How to profile memory and exeution time of functions?

Profile

If you need to get a better understanding of you functions memory and execution time, try these profiling options.

Q: How to read BUFR files with python?

BUFR

ECMWF created a python package that helps to read BUFR messages via ECCODES into pandas DataFrame.

Q: How to read ODB files with python?

ODB

ECMWF created a python package that helps to read ODB messages via ODC or pure Python into pandas DataFrame.

Q: How to retrieve data from MARS in python?

MARS

ECMWF has created a package that allows to retrieve data from MARS, easily.

Q: How to retrieve ERA5 from CDS and plot in Magics?

Magics

It is very easy to download and plot ERA5 data, using cdsapi and Magics.

Q: How to ignore user site packages?

When using python it is possible to have multiple site, where packages can be installed. the default is to use the path from the python interpreter and a user site.

Bash
1
2
3
4
5
6
7
$ python -c 'import sys;print("\n".join(sys.path))'

/jetfs/manual/enstools/v2021.11/lib/python38.zip
/jetfs/manual/enstools/v2021.11/lib/python3.8
/jetfs/manual/enstools/v2021.11/lib/python3.8/lib-dynload
/jetfs/home/USER/.local/lib/python3.8/site-packages
/jetfs/manual/enstools/v2021.11/lib/python3.8/site-packages

this shows that there is a user site in your HOME directory for that python version. Python will always look for a directory for it's own version.

You can disable that by altering the sys.path (python type list) manually or using an environment variable (export PYTHONNOUSERSITE=1).

Bash
1
2
3
4
5
6
7
8
9
export PYTHONNOUSERSITE=1
$ python -c 'import sys;print("\n".join(sys.path))'

/jetfs/manual/enstools/v2021.11/lib/python38.zip
/jetfs/manual/enstools/v2021.11/lib/python3.8
/jetfs/manual/enstools/v2021.11/lib/python3.8/lib-dynload
/jetfs/manual/enstools/v2021.11/lib/python3.8/site-packages
# unset the variable again
unset PYTHONNOUSERSITE
Last update: February 29, 2024
Created: November 9, 2022