Welcome to Jet¶

You have done it! Your Notebook Server is running on the Jet Cluster.

Welcome, please skip this, if your know where you are, as this is meant as an introduction. However, not complete as it is. If you need more infos about Notebooks have a look at my Homepage and there are numerous Tutorials on the Internet for any advice, but feel free to ask me and give me feedback.

Help can be found here:

• Goto Help -> JupyterLab Reference
• Goto Help -> Jupyter Reference
• Tour

In the following I will show you some useful things.

First: This Notebook is in your Home directory, but also on /jetfs/scratch for any reason.

Check your current Host¶

In [1]:

!hostnamectl

!hostnamectl

   Static hostname: localhost.localdomain
Transient hostname: jet03.jet.local
         Icon name: computer-server
           Chassis: server
  Operating System: ]8;;https://www.centos.org/CentOS Linux 8 (Core)]8;;
       CPE OS Name: cpe:/o:centos:centos:8
            Kernel: Linux 4.18.0-147.el8.x86_64
      Architecture: x86-64

Who is running this process¶

In [2]:

!whoami

!whoami

mblaschek

In [3]:

# Where are you
!pwd

# Where are you !pwd

/jetfs/home/mblaschek/Documents

In [4]:

# Show my Current Jobs in the Queue
!squeue -u $(id -u) -l

# Show my Current Jobs in the Queue !squeue -u $(id -u) -l

Fri Oct 16 23:10:58 2020
             JOBID PARTITION     NAME     USER    STATE       TIME TIME_LIMI  NODES NODELIST(REASON) 
               235   compute jupyters mblasche  RUNNING    6:29:07   8:00:00      1 jet03 

There you can see your TIME_LIMIT of 8 hours again and how much time has passed on your current jobs

In [5]:

# Import sys and check the executable (so what python you are using)
import sys
print(sys.executable)

# Import sys and check the executable (so what python you are using) import sys print(sys.executable)

/jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/bin/python

Notice the funny path?

Yes this is SPACK your nice scientific package manager. 
Allows to build custom libraries and applications tailored to our needs!
Please not that software currently installed in /opt on JET01 is not available here.
Software under /opt is deprecated and will be removed after a full transition 
to the current libraries

More information on SPACK

Check your environmental PATH Variable, could be useful to know¶

In [6]:

%env PATH

%env PATH

Out[6]:

'/jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/bin:/bin:/jetfs/spack/bin:/jetfs/userservices:/opt/jupyterhub/bin:/opt/slurm/bin:/sbin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/share/Modules/bin::/jetfs/home/mblaschek/bin:/jetfs/home/mblaschek/.local/bin'

Check what modules are currently loaded in this Notebook session¶

In [7]:

!module list --no-pager

!module list --no-pager

Currently Loaded Modulefiles:
 1) miniconda3/4.8.2-gcc-8.3.1-3m7b6t2  

Put this file load_modules_into_jupyter.conf into your Home directory and your Notebook environment will have these modules loaded. It is not possible to load these modules via magic (python magic ;). It needs the empty line at the end...

In [8]:

%%writefile ~/load_modules_into_jupyter.conf
eccodes/2.18.0-gcc-8.3.1-s7clum3

%%writefile ~/load_modules_into_jupyter.conf eccodes/2.18.0-gcc-8.3.1-s7clum3

Writing /jetfs/home/mblaschek/load_modules_into_jupyter.conf

In [9]:

!cat ~/load_modules_into_jupyter.conf

!cat ~/load_modules_into_jupyter.conf

eccodes/2.18.0-gcc-8.3.1-s7clum3

Restarting the kernel will show now loaded modules

In [1]:

# after restarting
!module list --no-pager

# after restarting !module list --no-pager

Currently Loaded Modulefiles:
 1) miniconda3/4.8.2-gcc-8.3.1-3m7b6t2        
 2) openmpi/4.0.5-gcc-8.3.1-773ztsv           
 3) hdf5/1.12.0-gcc-8.3.1-awl4atl             
 4) parallel-netcdf/1.12.1-gcc-8.3.1-xxrhtxn  
 5) netcdf-c/4.7.4-gcc-8.3.1-fh4nn6k          
 6) eccodes/2.18.0-gcc-8.3.1-s7clum3          

Currently available Modules¶

In [11]:

# --no-pager is needed only inside notebook
!module avail --no-pager

# --no-pager is needed only inside notebook !module avail --no-pager

--------- /jetfs/spack/share/spack/modules/linux-rhel8-skylake_avx512 ----------
anaconda2/2019.10-gcc-8.3.1-5pou6ji       
anaconda3/2019.10-gcc-8.3.1-tmy5mgp       
anaconda3/2020.07-gcc-8.3.1-weugqkf       
cdo/1.9.8-gcc-8.3.1-ipgvzeh               
eccodes/2.18.0-gcc-8.3.1-s7clum3          
hdf5/1.12.0-gcc-8.3.1-awl4atl             
intel-mkl/2020.3.279-gcc-8.3.1-5xeezjw    
miniconda2/4.7.12.1-gcc-8.3.1-zduqggv     
miniconda3/4.8.2-gcc-8.3.1-3m7b6t2        
netcdf-c/4.7.4-gcc-8.3.1-fh4nn6k          
netcdf-fortran/4.5.3-gcc-8.3.1-kfd2vkj    
openmpi/3.1.6-gcc-8.3.1-rk5av53           
openmpi/4.0.5-gcc-8.3.1-773ztsv           
parallel-netcdf/1.12.1-gcc-8.3.1-gng2jcu  
parallel-netcdf/1.12.1-gcc-8.3.1-xxrhtxn  

--------- /jetfs/spack/share/spack/modules/linux-rhel8-skylake_avx512 ----------
anaconda2/2019.10-gcc-8.3.1-5pou6ji       
anaconda3/2019.10-gcc-8.3.1-tmy5mgp       
anaconda3/2020.07-gcc-8.3.1-weugqkf       
cdo/1.9.8-gcc-8.3.1-ipgvzeh               
eccodes/2.18.0-gcc-8.3.1-s7clum3          
hdf5/1.12.0-gcc-8.3.1-awl4atl             
intel-mkl/2020.3.279-gcc-8.3.1-5xeezjw    
miniconda2/4.7.12.1-gcc-8.3.1-zduqggv     
miniconda3/4.8.2-gcc-8.3.1-3m7b6t2        
netcdf-c/4.7.4-gcc-8.3.1-fh4nn6k          
netcdf-fortran/4.5.3-gcc-8.3.1-kfd2vkj    
openmpi/3.1.6-gcc-8.3.1-rk5av53           
openmpi/4.0.5-gcc-8.3.1-773ztsv           
parallel-netcdf/1.12.1-gcc-8.3.1-gng2jcu  
parallel-netcdf/1.12.1-gcc-8.3.1-xxrhtxn  

Software on JET¶

Currently:

• Anaconda (2,3)
• Miniconda (2,3)
• OPENMPI (4.0.5, 3.6.1) optimized for JET
• HDF5 (1.12) optimized with OPENMPI
• NetCDF (Parallel (NC3), NC4->HDF5)
• Eccodes (2.18)
• CDO
• Intel compiled Libraries (License)
• Emoslib
• Ecaccess
• RTTOV

Currently there are a few choices:¶

1. Python (2, 3)
   • Anaconda
   • Miniconda
2. Julia (not yet)
3. R (not yet)

Install Python packages¶

In [12]:

# Please use the --user flag to install the packages into .local/lib/python...
%pip install --user numpy scipy

# Please use the --user flag to install the packages into .local/lib/python... %pip install --user numpy scipy

Requirement already satisfied: numpy in /jetfs/home/mblaschek/.local/lib/python3.8/site-packages (1.19.2)
Requirement already satisfied: scipy in /jetfs/home/mblaschek/.local/lib/python3.8/site-packages (1.5.2)
Note: you may need to restart the kernel to use updated packages.

In [13]:

import os
import sys
import site
# Need to update search Path / only necessary here for the first time
# sys.path.append(os.path.expandvars("$HOME/.local/lib/python3.8/site-packages"))
sys.path.append(site.getusersitepackages())

import os import sys import site # Need to update search Path / only necessary here for the first time # sys.path.append(os.path.expandvars("$HOME/.local/lib/python3.8/site-packages")) sys.path.append(site.getusersitepackages())

In [14]:

import numpy as np
import scipy as sp

# WORKS! ;)

import numpy as np import scipy as sp # WORKS! ;)

Using conda¶

In [15]:

!conda info

!conda info

     active environment : None
       user config file : /jetfs/home/mblaschek/.condarc
 populated config files : /jetfs/home/mblaschek/.condarc
          conda version : 4.8.5
    conda-build version : not installed
         python version : 3.8.1.final.0
       virtual packages : __glibc=2.28
       base environment : /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv  (read only)
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/pkgs
                          /jetfs/home/mblaschek/.conda/pkgs
       envs directories : /jetfs/home/mblaschek/.conda/envs
                          /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/envs
               platform : linux-64
             user-agent : conda/4.8.5 requests/2.22.0 CPython/3.8.1 Linux/4.18.0-147.el8.x86_64 centos/8.2.2004 glibc/2.28
                UID:GID : 54212:100
             netrc file : None
           offline mode : False

In [16]:

# Create a environment with conda
!conda create -y -n myenv python=3.7 numpy ipykernel

# Create a environment with conda !conda create -y -n myenv python=3.7 numpy ipykernel

Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /jetfs/home/mblaschek/.conda/envs/myenv

  added / updated specs:
    - ipykernel
    - numpy
    - python=3.7


The following NEW packages will be INSTALLED:

  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main
  backcall           pkgs/main/noarch::backcall-0.2.0-py_0
  blas               pkgs/main/linux-64::blas-1.0-mkl
  ca-certificates    pkgs/main/linux-64::ca-certificates-2020.10.14-0
  certifi            pkgs/main/linux-64::certifi-2020.6.20-py37_0
  decorator          pkgs/main/noarch::decorator-4.4.2-py_0
  intel-openmp       pkgs/main/linux-64::intel-openmp-2020.2-254
  ipykernel          pkgs/main/linux-64::ipykernel-5.3.4-py37h5ca1d4c_0
  ipython            pkgs/main/linux-64::ipython-7.18.1-py37h5ca1d4c_0
  ipython_genutils   pkgs/main/linux-64::ipython_genutils-0.2.0-py37_0
  jedi               pkgs/main/linux-64::jedi-0.17.2-py37_0
  jupyter_client     pkgs/main/noarch::jupyter_client-6.1.7-py_0
  jupyter_core       pkgs/main/linux-64::jupyter_core-4.6.3-py37_0
  ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.33.1-h53a641e_7
  libedit            pkgs/main/linux-64::libedit-3.1.20191231-h14c3975_1
  libffi             pkgs/main/linux-64::libffi-3.3-he6710b0_2
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-9.1.0-hdf63c60_0
  libsodium          pkgs/main/linux-64::libsodium-1.0.18-h7b6447c_0
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0
  mkl                pkgs/main/linux-64::mkl-2020.2-256
  mkl-service        pkgs/main/linux-64::mkl-service-2.3.0-py37he904b0f_0
  mkl_fft            pkgs/main/linux-64::mkl_fft-1.2.0-py37h23d657b_0
  mkl_random         pkgs/main/linux-64::mkl_random-1.1.1-py37h0573a6f_0
  ncurses            pkgs/main/linux-64::ncurses-6.2-he6710b0_1
  numpy              pkgs/main/linux-64::numpy-1.19.1-py37hbc911f0_0
  numpy-base         pkgs/main/linux-64::numpy-base-1.19.1-py37hfa32c7d_0
  openssl            pkgs/main/linux-64::openssl-1.1.1h-h7b6447c_0
  parso              pkgs/main/noarch::parso-0.7.0-py_0
  pexpect            pkgs/main/linux-64::pexpect-4.8.0-py37_1
  pickleshare        pkgs/main/linux-64::pickleshare-0.7.5-py37_1001
  pip                pkgs/main/linux-64::pip-20.2.3-py37_0
  prompt-toolkit     pkgs/main/noarch::prompt-toolkit-3.0.8-py_0
  ptyprocess         pkgs/main/linux-64::ptyprocess-0.6.0-py37_0
  pygments           pkgs/main/noarch::pygments-2.7.1-py_0
  python             pkgs/main/linux-64::python-3.7.9-h7579374_0
  python-dateutil    pkgs/main/noarch::python-dateutil-2.8.1-py_0
  pyzmq              pkgs/main/linux-64::pyzmq-19.0.2-py37he6710b0_1
  readline           pkgs/main/linux-64::readline-8.0-h7b6447c_0
  setuptools         pkgs/main/linux-64::setuptools-50.3.0-py37hb0f4dca_1
  six                pkgs/main/noarch::six-1.15.0-py_0
  sqlite             pkgs/main/linux-64::sqlite-3.33.0-h62c20be_0
  tk                 pkgs/main/linux-64::tk-8.6.10-hbc83047_0
  tornado            pkgs/main/linux-64::tornado-6.0.4-py37h7b6447c_1
  traitlets          pkgs/main/noarch::traitlets-5.0.5-py_0
  wcwidth            pkgs/main/noarch::wcwidth-0.2.5-py_0
  wheel              pkgs/main/noarch::wheel-0.35.1-py_0
  xz                 pkgs/main/linux-64::xz-5.2.5-h7b6447c_0
  zeromq             pkgs/main/linux-64::zeromq-4.3.3-he6710b0_3
  zlib               pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3


Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate myenv
#
# To deactivate an active environment, use
#
#     $ conda deactivate

Activating a conda environment¶

It is not recommended to use conda init bash, because it will write that specific version into your .bashrc file and will result in errors when you move to another module or newer version.

Hint: Checkout the guides on Gitlab

Load bash conda configuration¶

This script is used by conda init bash to set the configuration, we simply source (load) it and get the same functionality as we would be using conda init bash. (Do not use conda init bash)

source $(dirname $(which conda))/../etc/profile.d/conda.sh
# now activating the environment works
conda activate myenv

optionaly you can add this bash function to your .bashrc for convenience:

conda_activate(){
    if ! command -v conda &> /dev/null; then
        echo "Load ana/miniconda module first"
    else
        if [ $# -ne 1 ]; then
            echo "usage: conda_activate [env-name]"
            
        else
            source $(dirname $(which conda))/../etc/profile.d/conda.sh
            conda activate $1
        fi
    fi
}

After your .bashrc has been loaded (e.g. logout/login again) you should be able to use

module load anaconda3
conda_activate myenv

Activate your kernel in your notebooks¶

Please run the follwoing command and install the ipykernel, if it is not yet installed:

In [17]:

!~/.conda/envs/myenv/bin/python -m ipykernel install --user --name MYENV --display-name "MYCONDA"

!~/.conda/envs/myenv/bin/python -m ipykernel install --user --name MYENV --display-name "MYCONDA"

Installed kernelspec MYENV in /jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv

In [18]:

!jupyter kernelspec list

!jupyter kernelspec list

Available kernels:
  myenv      /jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv
  python3    /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/share/jupyter/kernels/python3

Custom Environment Python Environment¶

In [19]:

# Setup a virtual Environment for your Python in the directoy: myfancyenv
!python -m venv myfancyenv

# Setup a virtual Environment for your Python in the directoy: myfancyenv !python -m venv myfancyenv

In [20]:

# Install some package into that environment
# important is ipykernel to use it within jupyterlab/notebook
!./myfancyenv/bin/pip -q install numpy scipy pandas ipykernel

# Install some package into that environment # important is ipykernel to use it within jupyterlab/notebook !./myfancyenv/bin/pip -q install numpy scipy pandas ipykernel

WARNING: You are using pip version 19.2.3, however version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

In [21]:

# Allow this environment to be available on the LAUNCHERS (+) sign on the left
!./myfancyenv/bin/python -m ipykernel install --user --name fancy --display-name "My Python"

# Allow this environment to be available on the LAUNCHERS (+) sign on the left !./myfancyenv/bin/python -m ipykernel install --user --name fancy --display-name "My Python"

Installed kernelspec fancy in /jetfs/home/mblaschek/.local/share/jupyter/kernels/fancy

Wait a bit and the new python environment should be available on your launchers.

In [22]:

!jupyter kernelspec list

!jupyter kernelspec list

Available kernels:
  fancy      /jetfs/home/mblaschek/.local/share/jupyter/kernels/fancy
  myenv      /jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv
  python3    /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/share/jupyter/kernels/python3

There you go. Your personal Environment.

If you need help. Feel free to ask me. Enjoy!¶

---

Last update: January 26, 2023
Created: January 26, 2023