Welcome to Jet¶
You have done it! Your Notebook Server is running on the Jet Cluster.
Welcome, please skip this, if your know where you are, as this is meant as an introduction. However, not complete as it is. If you need more infos about Notebooks have a look at my Homepage and there are numerous Tutorials on the Internet for any advice, but feel free to ask me and give me feedback.
Help can be found here:
- Goto Help -> JupyterLab Reference
- Goto Help -> Jupyter Reference
- Tour
In the following I will show you some useful things.
First: This Notebook is in your Home directory, but also on /jetfs/scratch for any reason.
Check your current Host¶
In [1]:
Copied!
!hostnamectl
!hostnamectl
Static hostname: localhost.localdomain
Transient hostname: jet03.jet.local
Icon name: computer-server
Chassis: server
Operating System: ]8;;https://www.centos.org/CentOS Linux 8 (Core)]8;;
CPE OS Name: cpe:/o:centos:centos:8
Kernel: Linux 4.18.0-147.el8.x86_64
Architecture: x86-64
Who is running this process¶
In [2]:
Copied!
!whoami
!whoami
mblaschek
In [3]:
Copied!
# Where are you
!pwd
# Where are you
!pwd
/jetfs/home/mblaschek/Documents
In [4]:
Copied!
# Show my Current Jobs in the Queue
!squeue -u $(id -u) -l
# Show my Current Jobs in the Queue
!squeue -u $(id -u) -l
Fri Oct 16 23:10:58 2020
JOBID PARTITION NAME USER STATE TIME TIME_LIMI NODES NODELIST(REASON)
235 compute jupyters mblasche RUNNING 6:29:07 8:00:00 1 jet03
There you can see your TIME_LIMIT of 8 hours again and how much time has passed on your current jobs
In [5]:
Copied!
# Import sys and check the executable (so what python you are using)
import sys
print(sys.executable)
# Import sys and check the executable (so what python you are using)
import sys
print(sys.executable)
/jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/bin/python
Notice the funny path?
Yes this is SPACK your nice scientific package manager.
Allows to build custom libraries and applications tailored to our needs!
Please not that software currently installed in /opt on JET01 is not available here.
Software under /opt is deprecated and will be removed after a full transition
to the current librariesMore information on SPACK
Check your environmental PATH Variable, could be useful to know¶
In [6]:
Copied!
%env PATH
%env PATH
Out[6]:
'/jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/bin:/bin:/jetfs/spack/bin:/jetfs/userservices:/opt/jupyterhub/bin:/opt/slurm/bin:/sbin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/share/Modules/bin::/jetfs/home/mblaschek/bin:/jetfs/home/mblaschek/.local/bin'
Check what modules are currently loaded in this Notebook session¶
In [7]:
Copied!
!module list --no-pager
!module list --no-pager
Currently Loaded Modulefiles: 1) miniconda3/4.8.2-gcc-8.3.1-3m7b6t2
Put this file load_modules_into_jupyter.conf into your Home directory and your Notebook environment will have these modules loaded.
It is not possible to load these modules via magic (python magic ;).
It needs the empty line at the end...
In [8]:
Copied!
%%writefile ~/load_modules_into_jupyter.conf
eccodes/2.18.0-gcc-8.3.1-s7clum3
%%writefile ~/load_modules_into_jupyter.conf
eccodes/2.18.0-gcc-8.3.1-s7clum3
Writing /jetfs/home/mblaschek/load_modules_into_jupyter.conf
In [9]:
Copied!
!cat ~/load_modules_into_jupyter.conf
!cat ~/load_modules_into_jupyter.conf
eccodes/2.18.0-gcc-8.3.1-s7clum3
Restarting the kernel will show now loaded modules
In [1]:
Copied!
# after restarting
!module list --no-pager
# after restarting
!module list --no-pager
Currently Loaded Modulefiles: 1) miniconda3/4.8.2-gcc-8.3.1-3m7b6t2 2) openmpi/4.0.5-gcc-8.3.1-773ztsv 3) hdf5/1.12.0-gcc-8.3.1-awl4atl 4) parallel-netcdf/1.12.1-gcc-8.3.1-xxrhtxn 5) netcdf-c/4.7.4-gcc-8.3.1-fh4nn6k 6) eccodes/2.18.0-gcc-8.3.1-s7clum3
Currently available Modules¶
In [11]:
Copied!
# --no-pager is needed only inside notebook
!module avail --no-pager
# --no-pager is needed only inside notebook
!module avail --no-pager
--------- /jetfs/spack/share/spack/modules/linux-rhel8-skylake_avx512 ---------- anaconda2/2019.10-gcc-8.3.1-5pou6ji anaconda3/2019.10-gcc-8.3.1-tmy5mgp anaconda3/2020.07-gcc-8.3.1-weugqkf cdo/1.9.8-gcc-8.3.1-ipgvzeh eccodes/2.18.0-gcc-8.3.1-s7clum3 hdf5/1.12.0-gcc-8.3.1-awl4atl intel-mkl/2020.3.279-gcc-8.3.1-5xeezjw miniconda2/4.7.12.1-gcc-8.3.1-zduqggv miniconda3/4.8.2-gcc-8.3.1-3m7b6t2 netcdf-c/4.7.4-gcc-8.3.1-fh4nn6k netcdf-fortran/4.5.3-gcc-8.3.1-kfd2vkj openmpi/3.1.6-gcc-8.3.1-rk5av53 openmpi/4.0.5-gcc-8.3.1-773ztsv parallel-netcdf/1.12.1-gcc-8.3.1-gng2jcu parallel-netcdf/1.12.1-gcc-8.3.1-xxrhtxn --------- /jetfs/spack/share/spack/modules/linux-rhel8-skylake_avx512 ---------- anaconda2/2019.10-gcc-8.3.1-5pou6ji anaconda3/2019.10-gcc-8.3.1-tmy5mgp anaconda3/2020.07-gcc-8.3.1-weugqkf cdo/1.9.8-gcc-8.3.1-ipgvzeh eccodes/2.18.0-gcc-8.3.1-s7clum3 hdf5/1.12.0-gcc-8.3.1-awl4atl intel-mkl/2020.3.279-gcc-8.3.1-5xeezjw miniconda2/4.7.12.1-gcc-8.3.1-zduqggv miniconda3/4.8.2-gcc-8.3.1-3m7b6t2 netcdf-c/4.7.4-gcc-8.3.1-fh4nn6k netcdf-fortran/4.5.3-gcc-8.3.1-kfd2vkj openmpi/3.1.6-gcc-8.3.1-rk5av53 openmpi/4.0.5-gcc-8.3.1-773ztsv parallel-netcdf/1.12.1-gcc-8.3.1-gng2jcu parallel-netcdf/1.12.1-gcc-8.3.1-xxrhtxn
Software on JET¶
Currently:
- Anaconda (2,3)
- Miniconda (2,3)
- OPENMPI (4.0.5, 3.6.1) optimized for JET
- HDF5 (1.12) optimized with OPENMPI
- NetCDF (Parallel (NC3), NC4->HDF5)
- Eccodes (2.18)
- CDO
- Intel compiled Libraries (License)
- Emoslib
- Ecaccess
- RTTOV
Currently there are a few choices:¶
- Python (2, 3)
- Anaconda
- Miniconda
- Julia (not yet)
- R (not yet)
Install Python packages¶
In [12]:
Copied!
# Please use the --user flag to install the packages into .local/lib/python...
%pip install --user numpy scipy
# Please use the --user flag to install the packages into .local/lib/python...
%pip install --user numpy scipy
Requirement already satisfied: numpy in /jetfs/home/mblaschek/.local/lib/python3.8/site-packages (1.19.2) Requirement already satisfied: scipy in /jetfs/home/mblaschek/.local/lib/python3.8/site-packages (1.5.2) Note: you may need to restart the kernel to use updated packages.
In [13]:
Copied!
import os
import sys
import site
# Need to update search Path / only necessary here for the first time
# sys.path.append(os.path.expandvars("$HOME/.local/lib/python3.8/site-packages"))
sys.path.append(site.getusersitepackages())
import os
import sys
import site
# Need to update search Path / only necessary here for the first time
# sys.path.append(os.path.expandvars("$HOME/.local/lib/python3.8/site-packages"))
sys.path.append(site.getusersitepackages())
In [14]:
Copied!
import numpy as np
import scipy as sp
# WORKS! ;)
import numpy as np
import scipy as sp
# WORKS! ;)
Using conda¶
In [15]:
Copied!
!conda info
!conda info
active environment : None
user config file : /jetfs/home/mblaschek/.condarc
populated config files : /jetfs/home/mblaschek/.condarc
conda version : 4.8.5
conda-build version : not installed
python version : 3.8.1.final.0
virtual packages : __glibc=2.28
base environment : /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv (read only)
channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/pkgs
/jetfs/home/mblaschek/.conda/pkgs
envs directories : /jetfs/home/mblaschek/.conda/envs
/jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/envs
platform : linux-64
user-agent : conda/4.8.5 requests/2.22.0 CPython/3.8.1 Linux/4.18.0-147.el8.x86_64 centos/8.2.2004 glibc/2.28
UID:GID : 54212:100
netrc file : None
offline mode : False
In [16]:
Copied!
# Create a environment with conda
!conda create -y -n myenv python=3.7 numpy ipykernel
# Create a environment with conda
!conda create -y -n myenv python=3.7 numpy ipykernel
Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: done
## Package Plan ##
environment location: /jetfs/home/mblaschek/.conda/envs/myenv
added / updated specs:
- ipykernel
- numpy
- python=3.7
The following NEW packages will be INSTALLED:
_libgcc_mutex pkgs/main/linux-64::_libgcc_mutex-0.1-main
backcall pkgs/main/noarch::backcall-0.2.0-py_0
blas pkgs/main/linux-64::blas-1.0-mkl
ca-certificates pkgs/main/linux-64::ca-certificates-2020.10.14-0
certifi pkgs/main/linux-64::certifi-2020.6.20-py37_0
decorator pkgs/main/noarch::decorator-4.4.2-py_0
intel-openmp pkgs/main/linux-64::intel-openmp-2020.2-254
ipykernel pkgs/main/linux-64::ipykernel-5.3.4-py37h5ca1d4c_0
ipython pkgs/main/linux-64::ipython-7.18.1-py37h5ca1d4c_0
ipython_genutils pkgs/main/linux-64::ipython_genutils-0.2.0-py37_0
jedi pkgs/main/linux-64::jedi-0.17.2-py37_0
jupyter_client pkgs/main/noarch::jupyter_client-6.1.7-py_0
jupyter_core pkgs/main/linux-64::jupyter_core-4.6.3-py37_0
ld_impl_linux-64 pkgs/main/linux-64::ld_impl_linux-64-2.33.1-h53a641e_7
libedit pkgs/main/linux-64::libedit-3.1.20191231-h14c3975_1
libffi pkgs/main/linux-64::libffi-3.3-he6710b0_2
libgcc-ng pkgs/main/linux-64::libgcc-ng-9.1.0-hdf63c60_0
libsodium pkgs/main/linux-64::libsodium-1.0.18-h7b6447c_0
libstdcxx-ng pkgs/main/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0
mkl pkgs/main/linux-64::mkl-2020.2-256
mkl-service pkgs/main/linux-64::mkl-service-2.3.0-py37he904b0f_0
mkl_fft pkgs/main/linux-64::mkl_fft-1.2.0-py37h23d657b_0
mkl_random pkgs/main/linux-64::mkl_random-1.1.1-py37h0573a6f_0
ncurses pkgs/main/linux-64::ncurses-6.2-he6710b0_1
numpy pkgs/main/linux-64::numpy-1.19.1-py37hbc911f0_0
numpy-base pkgs/main/linux-64::numpy-base-1.19.1-py37hfa32c7d_0
openssl pkgs/main/linux-64::openssl-1.1.1h-h7b6447c_0
parso pkgs/main/noarch::parso-0.7.0-py_0
pexpect pkgs/main/linux-64::pexpect-4.8.0-py37_1
pickleshare pkgs/main/linux-64::pickleshare-0.7.5-py37_1001
pip pkgs/main/linux-64::pip-20.2.3-py37_0
prompt-toolkit pkgs/main/noarch::prompt-toolkit-3.0.8-py_0
ptyprocess pkgs/main/linux-64::ptyprocess-0.6.0-py37_0
pygments pkgs/main/noarch::pygments-2.7.1-py_0
python pkgs/main/linux-64::python-3.7.9-h7579374_0
python-dateutil pkgs/main/noarch::python-dateutil-2.8.1-py_0
pyzmq pkgs/main/linux-64::pyzmq-19.0.2-py37he6710b0_1
readline pkgs/main/linux-64::readline-8.0-h7b6447c_0
setuptools pkgs/main/linux-64::setuptools-50.3.0-py37hb0f4dca_1
six pkgs/main/noarch::six-1.15.0-py_0
sqlite pkgs/main/linux-64::sqlite-3.33.0-h62c20be_0
tk pkgs/main/linux-64::tk-8.6.10-hbc83047_0
tornado pkgs/main/linux-64::tornado-6.0.4-py37h7b6447c_1
traitlets pkgs/main/noarch::traitlets-5.0.5-py_0
wcwidth pkgs/main/noarch::wcwidth-0.2.5-py_0
wheel pkgs/main/noarch::wheel-0.35.1-py_0
xz pkgs/main/linux-64::xz-5.2.5-h7b6447c_0
zeromq pkgs/main/linux-64::zeromq-4.3.3-he6710b0_3
zlib pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
# $ conda activate myenv
#
# To deactivate an active environment, use
#
# $ conda deactivate
Activating a conda environment¶
It is not recommended to use conda init bash, because it will write that specific version into your .bashrc file and will result in errors when you move to another module or newer version.
Hint: Checkout the guides on Gitlab
Load bash conda configuration¶
This script is used by conda init bash to set the configuration, we simply source (load) it and get the same functionality as we would be using conda init bash. (Do not use conda init bash)
source $(dirname $(which conda))/../etc/profile.d/conda.sh
# now activating the environment works
conda activate myenv
optionaly you can add this bash function to your .bashrc for convenience:
conda_activate(){
if ! command -v conda &> /dev/null; then
echo "Load ana/miniconda module first"
else
if [ $# -ne 1 ]; then
echo "usage: conda_activate [env-name]"
else
source $(dirname $(which conda))/../etc/profile.d/conda.sh
conda activate $1
fi
fi
}
After your .bashrc has been loaded (e.g. logout/login again) you should be able to use
module load anaconda3
conda_activate myenv
Activate your kernel in your notebooks¶
Please run the follwoing command and install the ipykernel, if it is not yet installed:
In [17]:
Copied!
!~/.conda/envs/myenv/bin/python -m ipykernel install --user --name MYENV --display-name "MYCONDA"
!~/.conda/envs/myenv/bin/python -m ipykernel install --user --name MYENV --display-name "MYCONDA"
Installed kernelspec MYENV in /jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv
In [18]:
Copied!
!jupyter kernelspec list
!jupyter kernelspec list
Available kernels: myenv /jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv python3 /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/share/jupyter/kernels/python3
Custom Environment Python Environment¶
In [19]:
Copied!
# Setup a virtual Environment for your Python in the directoy: myfancyenv
!python -m venv myfancyenv
# Setup a virtual Environment for your Python in the directoy: myfancyenv
!python -m venv myfancyenv
In [20]:
Copied!
# Install some package into that environment
# important is ipykernel to use it within jupyterlab/notebook
!./myfancyenv/bin/pip -q install numpy scipy pandas ipykernel
# Install some package into that environment
# important is ipykernel to use it within jupyterlab/notebook
!./myfancyenv/bin/pip -q install numpy scipy pandas ipykernel
WARNING: You are using pip version 19.2.3, however version 20.2.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
In [21]:
Copied!
# Allow this environment to be available on the LAUNCHERS (+) sign on the left
!./myfancyenv/bin/python -m ipykernel install --user --name fancy --display-name "My Python"
# Allow this environment to be available on the LAUNCHERS (+) sign on the left
!./myfancyenv/bin/python -m ipykernel install --user --name fancy --display-name "My Python"
Installed kernelspec fancy in /jetfs/home/mblaschek/.local/share/jupyter/kernels/fancy
Wait a bit and the new python environment should be available on your launchers.
In [22]:
Copied!
!jupyter kernelspec list
!jupyter kernelspec list
Available kernels: fancy /jetfs/home/mblaschek/.local/share/jupyter/kernels/fancy myenv /jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv python3 /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/share/jupyter/kernels/python3
There you go. Your personal Environment.