Setting up a user environment¶

Topics

• Setup a user environment using conda
  • Add user environment to notebook kernels
  • Activate your conda environment
• Setup a user environment using venv
• Include PATH in Notebook Kernel

In [1]:

!module list

!module list

Currently Loaded Modulefiles:
 1) miniconda3/4.8.2-gcc-8.3.1-3m7b6t2  
>

Using conda¶

In [2]:

!conda info

!conda info

     active environment : None
       user config file : /jetfs/home/mblaschek/.condarc
 populated config files : /jetfs/home/mblaschek/.condarc
          conda version : 4.8.5
    conda-build version : not installed
         python version : 3.8.1.final.0
       virtual packages : __glibc=2.28
       base environment : /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv  (read only)
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/pkgs
                          /jetfs/home/mblaschek/.conda/pkgs
       envs directories : /jetfs/home/mblaschek/.conda/envs
                          /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/envs
               platform : linux-64
             user-agent : conda/4.8.5 requests/2.22.0 CPython/3.8.1 Linux/4.18.0-147.el8.x86_64 centos/8.2.2004 glibc/2.28
                UID:GID : 54212:100
             netrc file : None
           offline mode : False

In [3]:

# Create a environment with conda
!conda create -y -n myenv python=3.7 numpy ipykernel

# Create a environment with conda !conda create -y -n myenv python=3.7 numpy ipykernel

Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.8.5
  latest version: 4.9.2

Please update conda by running

    $ conda update -n base -c defaults conda



## Package Plan ##

  environment location: /jetfs/home/mblaschek/.conda/envs/myenv

  added / updated specs:
    - ipykernel
    - numpy
    - python=3.7


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    certifi-2020.11.8          |   py37h06a4308_0         148 KB
    ipython-7.19.0             |   py37hb070fc8_0         978 KB
    jupyter_core-4.7.0         |   py37h06a4308_0          68 KB
    numpy-1.19.2               |   py37h54aff64_0          22 KB
    numpy-base-1.19.2          |   py37hfa32c7d_0         4.1 MB
    pexpect-4.8.0              |     pyhd3eb1b0_3          53 KB
    pip-20.2.4                 |   py37h06a4308_0         1.7 MB
    ptyprocess-0.6.0           |     pyhd3eb1b0_2          20 KB
    pygments-2.7.2             |     pyhd3eb1b0_0         674 KB
    setuptools-50.3.1          |   py37h06a4308_1         711 KB
    six-1.15.0                 |   py37h06a4308_0          27 KB
    wheel-0.35.1               |     pyhd3eb1b0_0          38 KB
    ------------------------------------------------------------
                                           Total:         8.5 MB

The following NEW packages will be INSTALLED:

  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main
  backcall           pkgs/main/noarch::backcall-0.2.0-py_0
  blas               pkgs/main/linux-64::blas-1.0-mkl
  ca-certificates    pkgs/main/linux-64::ca-certificates-2020.10.14-0
  certifi            pkgs/main/linux-64::certifi-2020.11.8-py37h06a4308_0
  decorator          pkgs/main/noarch::decorator-4.4.2-py_0
  intel-openmp       pkgs/main/linux-64::intel-openmp-2020.2-254
  ipykernel          pkgs/main/linux-64::ipykernel-5.3.4-py37h5ca1d4c_0
  ipython            pkgs/main/linux-64::ipython-7.19.0-py37hb070fc8_0
  ipython_genutils   pkgs/main/linux-64::ipython_genutils-0.2.0-py37_0
  jedi               pkgs/main/linux-64::jedi-0.17.2-py37_0
  jupyter_client     pkgs/main/noarch::jupyter_client-6.1.7-py_0
  jupyter_core       pkgs/main/linux-64::jupyter_core-4.7.0-py37h06a4308_0
  ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.33.1-h53a641e_7
  libedit            pkgs/main/linux-64::libedit-3.1.20191231-h14c3975_1
  libffi             pkgs/main/linux-64::libffi-3.3-he6710b0_2
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-9.1.0-hdf63c60_0
  libsodium          pkgs/main/linux-64::libsodium-1.0.18-h7b6447c_0
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0
  mkl                pkgs/main/linux-64::mkl-2020.2-256
  mkl-service        pkgs/main/linux-64::mkl-service-2.3.0-py37he904b0f_0
  mkl_fft            pkgs/main/linux-64::mkl_fft-1.2.0-py37h23d657b_0
  mkl_random         pkgs/main/linux-64::mkl_random-1.1.1-py37h0573a6f_0
  ncurses            pkgs/main/linux-64::ncurses-6.2-he6710b0_1
  numpy              pkgs/main/linux-64::numpy-1.19.2-py37h54aff64_0
  numpy-base         pkgs/main/linux-64::numpy-base-1.19.2-py37hfa32c7d_0
  openssl            pkgs/main/linux-64::openssl-1.1.1h-h7b6447c_0
  parso              pkgs/main/noarch::parso-0.7.0-py_0
  pexpect            pkgs/main/noarch::pexpect-4.8.0-pyhd3eb1b0_3
  pickleshare        pkgs/main/linux-64::pickleshare-0.7.5-py37_1001
  pip                pkgs/main/linux-64::pip-20.2.4-py37h06a4308_0
  prompt-toolkit     pkgs/main/noarch::prompt-toolkit-3.0.8-py_0
  ptyprocess         pkgs/main/noarch::ptyprocess-0.6.0-pyhd3eb1b0_2
  pygments           pkgs/main/noarch::pygments-2.7.2-pyhd3eb1b0_0
  python             pkgs/main/linux-64::python-3.7.9-h7579374_0
  python-dateutil    pkgs/main/noarch::python-dateutil-2.8.1-py_0
  pyzmq              pkgs/main/linux-64::pyzmq-19.0.2-py37he6710b0_1
  readline           pkgs/main/linux-64::readline-8.0-h7b6447c_0
  setuptools         pkgs/main/linux-64::setuptools-50.3.1-py37h06a4308_1
  six                pkgs/main/linux-64::six-1.15.0-py37h06a4308_0
  sqlite             pkgs/main/linux-64::sqlite-3.33.0-h62c20be_0
  tk                 pkgs/main/linux-64::tk-8.6.10-hbc83047_0
  tornado            pkgs/main/linux-64::tornado-6.0.4-py37h7b6447c_1
  traitlets          pkgs/main/noarch::traitlets-5.0.5-py_0
  wcwidth            pkgs/main/noarch::wcwidth-0.2.5-py_0
  wheel              pkgs/main/noarch::wheel-0.35.1-pyhd3eb1b0_0
  xz                 pkgs/main/linux-64::xz-5.2.5-h7b6447c_0
  zeromq             pkgs/main/linux-64::zeromq-4.3.3-he6710b0_3
  zlib               pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3



Downloading and Extracting Packages
numpy-1.19.2         | 22 KB     | ##################################### | 100% 
pip-20.2.4           | 1.7 MB    | ##################################### | 100% 
numpy-base-1.19.2    | 4.1 MB    | ##################################### | 100% 
certifi-2020.11.8    | 148 KB    | ##################################### | 100% 
wheel-0.35.1         | 38 KB     | ##################################### | 100% 
ipython-7.19.0       | 978 KB    | ##################################### | 100% 
pexpect-4.8.0        | 53 KB     | ##################################### | 100% 
pygments-2.7.2       | 674 KB    | ##################################### | 100% 
ptyprocess-0.6.0     | 20 KB     | ##################################### | 100% 
jupyter_core-4.7.0   | 68 KB     | ##################################### | 100% 
setuptools-50.3.1    | 711 KB    | ##################################### | 100% 
six-1.15.0           | 27 KB     | ##################################### | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate myenv
#
# To deactivate an active environment, use
#
#     $ conda deactivate

Add the new environment to your active kernels¶

This way you can choose the kernel from the Jupyterlab launcher.

In [4]:

!~/.conda/envs/myenv/bin/python -m ipykernel install --user --name MYENV --display-name "Py3.7"

!~/.conda/envs/myenv/bin/python -m ipykernel install --user --name MYENV --display-name "Py3.7"

Installed kernelspec MYENV in /jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv

Activate your conda environment¶

Conda Environments need to be added to the PATH to work well. So there is an activate function for these conda environments.

However, running

conda init bash

will edit your .bashrc and that is not nice, especially when you might move to another module. Therefore this is not recommended to do.

This script is used by conda init bash to set the configuration, we simply source (load) it and get the same functionality as we would be using conda init bash.

(Do not use conda init bash)

# source the conda functionality
eval "$(conda shell.bash hook)"
# or
source $(dirname $(which conda))/../etc/profile.d/conda.sh
# now activating the environment works
conda activate myenv

bash conda_activate function¶

optionally you can add this bash function to your .bashrc for convenience:

conda_activate(){
    if ! command -v conda &> /dev/null; then
        echo "Load ana/miniconda module first"
    else
        if [ $# -ne 1 ]; then
            echo "usage: conda_activate [env-name]"
            
        else
            source $(dirname $(which conda))/../etc/profile.d/conda.sh
            conda activate $1
        fi
    fi
}

Source your bashrc by running: source ~/.bashrc or logout/login again. Then you should be able to use the function:

module load anaconda3
conda_activate myenv

In [5]:

# List available Kernels
!jupyter kernelspec list

# List available Kernels !jupyter kernelspec list

Available kernels:
  myenv      /jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv
  python3    /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/share/jupyter/kernels/python3

Using virtual environment¶

In [6]:

# Setup a virtual Environment for your Python in the directoy: myfancyenv
!python -m venv myfancyenv

# Setup a virtual Environment for your Python in the directoy: myfancyenv !python -m venv myfancyenv

In [7]:

# Creates a directory with new environment
!ls myfancyenv/

# Creates a directory with new environment !ls myfancyenv/

bin  include  lib  lib64  pyvenv.cfg

In [8]:

# Install some package into that environment
# important is ipykernel to use it within jupyterlab/notebook
!./myfancyenv/bin/pip -q install numpy scipy pandas ipykernel

# Install some package into that environment # important is ipykernel to use it within jupyterlab/notebook !./myfancyenv/bin/pip -q install numpy scipy pandas ipykernel

WARNING: You are using pip version 19.2.3, however version 20.2.4 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

In [9]:

# Allow this environment to be available on the LAUNCHERS (+) sign on the left
!./myfancyenv/bin/python -m ipykernel install --user --name fancy --display-name "My Python"

# Allow this environment to be available on the LAUNCHERS (+) sign on the left !./myfancyenv/bin/python -m ipykernel install --user --name fancy --display-name "My Python"

Installed kernelspec fancy in /jetfs/home/mblaschek/.local/share/jupyter/kernels/fancy

Wait a bit and the new python environment should be available on your launchers.

In [10]:

!jupyter kernelspec list

!jupyter kernelspec list

Available kernels:
  fancy      /jetfs/home/mblaschek/.local/share/jupyter/kernels/fancy
  myenv      /jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv
  python3    /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/share/jupyter/kernels/python3

Add PATH to your user environment¶

In order to get full access to all tools installed in your local environment, add it to your PATH.

Open your kernel.json file located /jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv/kernel.json:

{
    "argv": [
        "/jetfs/home/mblaschek/.conda/envs/myenv/bin/python",
        "-m",
        "ipykernel_launcher",
        "-f",
        "{connection_file}"
    ],
    "env" : {
        "PATH": "/jetfs/home/mblaschek/.conda/envs/myenv/bin/:${PATH}",
        "LD_LIBRARY_PATH": "/jetfs/home/mblaschek/.conda/envs/myenv/lib:${LD_LIBRARY_PATH}"
    },
    "display_name": "MYENV",
    "language": "python"
}

Finished¶

---

Last update: December 12, 2022
Created: October 14, 2021