Setting up a user environment¶
Topics
- Setup a user environment using conda
- Setup a user environment using venv
- Include
PATHin Notebook Kernel
In [1]:
Copied!
!module list
!module list
Currently Loaded Modulefiles: 1) miniconda3/4.8.2-gcc-8.3.1-3m7b6t2 >
Using conda¶
In [2]:
Copied!
!conda info
!conda info
active environment : None
user config file : /jetfs/home/mblaschek/.condarc
populated config files : /jetfs/home/mblaschek/.condarc
conda version : 4.8.5
conda-build version : not installed
python version : 3.8.1.final.0
virtual packages : __glibc=2.28
base environment : /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv (read only)
channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/pkgs
/jetfs/home/mblaschek/.conda/pkgs
envs directories : /jetfs/home/mblaschek/.conda/envs
/jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/envs
platform : linux-64
user-agent : conda/4.8.5 requests/2.22.0 CPython/3.8.1 Linux/4.18.0-147.el8.x86_64 centos/8.2.2004 glibc/2.28
UID:GID : 54212:100
netrc file : None
offline mode : False
In [3]:
Copied!
# Create a environment with conda
!conda create -y -n myenv python=3.7 numpy ipykernel
# Create a environment with conda
!conda create -y -n myenv python=3.7 numpy ipykernel
Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: done
==> WARNING: A newer version of conda exists. <==
current version: 4.8.5
latest version: 4.9.2
Please update conda by running
$ conda update -n base -c defaults conda
## Package Plan ##
environment location: /jetfs/home/mblaschek/.conda/envs/myenv
added / updated specs:
- ipykernel
- numpy
- python=3.7
The following packages will be downloaded:
package | build
---------------------------|-----------------
certifi-2020.11.8 | py37h06a4308_0 148 KB
ipython-7.19.0 | py37hb070fc8_0 978 KB
jupyter_core-4.7.0 | py37h06a4308_0 68 KB
numpy-1.19.2 | py37h54aff64_0 22 KB
numpy-base-1.19.2 | py37hfa32c7d_0 4.1 MB
pexpect-4.8.0 | pyhd3eb1b0_3 53 KB
pip-20.2.4 | py37h06a4308_0 1.7 MB
ptyprocess-0.6.0 | pyhd3eb1b0_2 20 KB
pygments-2.7.2 | pyhd3eb1b0_0 674 KB
setuptools-50.3.1 | py37h06a4308_1 711 KB
six-1.15.0 | py37h06a4308_0 27 KB
wheel-0.35.1 | pyhd3eb1b0_0 38 KB
------------------------------------------------------------
Total: 8.5 MB
The following NEW packages will be INSTALLED:
_libgcc_mutex pkgs/main/linux-64::_libgcc_mutex-0.1-main
backcall pkgs/main/noarch::backcall-0.2.0-py_0
blas pkgs/main/linux-64::blas-1.0-mkl
ca-certificates pkgs/main/linux-64::ca-certificates-2020.10.14-0
certifi pkgs/main/linux-64::certifi-2020.11.8-py37h06a4308_0
decorator pkgs/main/noarch::decorator-4.4.2-py_0
intel-openmp pkgs/main/linux-64::intel-openmp-2020.2-254
ipykernel pkgs/main/linux-64::ipykernel-5.3.4-py37h5ca1d4c_0
ipython pkgs/main/linux-64::ipython-7.19.0-py37hb070fc8_0
ipython_genutils pkgs/main/linux-64::ipython_genutils-0.2.0-py37_0
jedi pkgs/main/linux-64::jedi-0.17.2-py37_0
jupyter_client pkgs/main/noarch::jupyter_client-6.1.7-py_0
jupyter_core pkgs/main/linux-64::jupyter_core-4.7.0-py37h06a4308_0
ld_impl_linux-64 pkgs/main/linux-64::ld_impl_linux-64-2.33.1-h53a641e_7
libedit pkgs/main/linux-64::libedit-3.1.20191231-h14c3975_1
libffi pkgs/main/linux-64::libffi-3.3-he6710b0_2
libgcc-ng pkgs/main/linux-64::libgcc-ng-9.1.0-hdf63c60_0
libsodium pkgs/main/linux-64::libsodium-1.0.18-h7b6447c_0
libstdcxx-ng pkgs/main/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0
mkl pkgs/main/linux-64::mkl-2020.2-256
mkl-service pkgs/main/linux-64::mkl-service-2.3.0-py37he904b0f_0
mkl_fft pkgs/main/linux-64::mkl_fft-1.2.0-py37h23d657b_0
mkl_random pkgs/main/linux-64::mkl_random-1.1.1-py37h0573a6f_0
ncurses pkgs/main/linux-64::ncurses-6.2-he6710b0_1
numpy pkgs/main/linux-64::numpy-1.19.2-py37h54aff64_0
numpy-base pkgs/main/linux-64::numpy-base-1.19.2-py37hfa32c7d_0
openssl pkgs/main/linux-64::openssl-1.1.1h-h7b6447c_0
parso pkgs/main/noarch::parso-0.7.0-py_0
pexpect pkgs/main/noarch::pexpect-4.8.0-pyhd3eb1b0_3
pickleshare pkgs/main/linux-64::pickleshare-0.7.5-py37_1001
pip pkgs/main/linux-64::pip-20.2.4-py37h06a4308_0
prompt-toolkit pkgs/main/noarch::prompt-toolkit-3.0.8-py_0
ptyprocess pkgs/main/noarch::ptyprocess-0.6.0-pyhd3eb1b0_2
pygments pkgs/main/noarch::pygments-2.7.2-pyhd3eb1b0_0
python pkgs/main/linux-64::python-3.7.9-h7579374_0
python-dateutil pkgs/main/noarch::python-dateutil-2.8.1-py_0
pyzmq pkgs/main/linux-64::pyzmq-19.0.2-py37he6710b0_1
readline pkgs/main/linux-64::readline-8.0-h7b6447c_0
setuptools pkgs/main/linux-64::setuptools-50.3.1-py37h06a4308_1
six pkgs/main/linux-64::six-1.15.0-py37h06a4308_0
sqlite pkgs/main/linux-64::sqlite-3.33.0-h62c20be_0
tk pkgs/main/linux-64::tk-8.6.10-hbc83047_0
tornado pkgs/main/linux-64::tornado-6.0.4-py37h7b6447c_1
traitlets pkgs/main/noarch::traitlets-5.0.5-py_0
wcwidth pkgs/main/noarch::wcwidth-0.2.5-py_0
wheel pkgs/main/noarch::wheel-0.35.1-pyhd3eb1b0_0
xz pkgs/main/linux-64::xz-5.2.5-h7b6447c_0
zeromq pkgs/main/linux-64::zeromq-4.3.3-he6710b0_3
zlib pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3
Downloading and Extracting Packages
numpy-1.19.2 | 22 KB | ##################################### | 100%
pip-20.2.4 | 1.7 MB | ##################################### | 100%
numpy-base-1.19.2 | 4.1 MB | ##################################### | 100%
certifi-2020.11.8 | 148 KB | ##################################### | 100%
wheel-0.35.1 | 38 KB | ##################################### | 100%
ipython-7.19.0 | 978 KB | ##################################### | 100%
pexpect-4.8.0 | 53 KB | ##################################### | 100%
pygments-2.7.2 | 674 KB | ##################################### | 100%
ptyprocess-0.6.0 | 20 KB | ##################################### | 100%
jupyter_core-4.7.0 | 68 KB | ##################################### | 100%
setuptools-50.3.1 | 711 KB | ##################################### | 100%
six-1.15.0 | 27 KB | ##################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
# $ conda activate myenv
#
# To deactivate an active environment, use
#
# $ conda deactivate
Add the new environment to your active kernels¶
This way you can choose the kernel from the Jupyterlab launcher.
In [4]:
Copied!
!~/.conda/envs/myenv/bin/python -m ipykernel install --user --name MYENV --display-name "Py3.7"
!~/.conda/envs/myenv/bin/python -m ipykernel install --user --name MYENV --display-name "Py3.7"
Installed kernelspec MYENV in /jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv
Activate your conda environment¶
Conda Environments need to be added to the PATH to work well. So there is an activate function for these conda environments.
However, running
conda init bash
will edit your .bashrc and that is not nice, especially when you might move to another module.
Therefore this is not recommended to do.
This script is used by conda init bash to set the configuration, we simply source (load) it and get the same functionality as we would be using conda init bash.
(Do not use conda init bash)
# source the conda functionality
eval "$(conda shell.bash hook)"
# or
source $(dirname $(which conda))/../etc/profile.d/conda.sh
# now activating the environment works
conda activate myenv
bash conda_activate function¶
optionally you can add this bash function to your .bashrc for convenience:
conda_activate(){
if ! command -v conda &> /dev/null; then
echo "Load ana/miniconda module first"
else
if [ $# -ne 1 ]; then
echo "usage: conda_activate [env-name]"
else
source $(dirname $(which conda))/../etc/profile.d/conda.sh
conda activate $1
fi
fi
}
Source your bashrc by running: source ~/.bashrc or logout/login again. Then you should be able to use the function:
module load anaconda3
conda_activate myenv
In [5]:
Copied!
# List available Kernels
!jupyter kernelspec list
# List available Kernels
!jupyter kernelspec list
Available kernels: myenv /jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv python3 /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/share/jupyter/kernels/python3
Using virtual environment¶
In [6]:
Copied!
# Setup a virtual Environment for your Python in the directoy: myfancyenv
!python -m venv myfancyenv
# Setup a virtual Environment for your Python in the directoy: myfancyenv
!python -m venv myfancyenv
In [7]:
Copied!
# Creates a directory with new environment
!ls myfancyenv/
# Creates a directory with new environment
!ls myfancyenv/
bin include lib lib64 pyvenv.cfg
In [8]:
Copied!
# Install some package into that environment
# important is ipykernel to use it within jupyterlab/notebook
!./myfancyenv/bin/pip -q install numpy scipy pandas ipykernel
# Install some package into that environment
# important is ipykernel to use it within jupyterlab/notebook
!./myfancyenv/bin/pip -q install numpy scipy pandas ipykernel
WARNING: You are using pip version 19.2.3, however version 20.2.4 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
In [9]:
Copied!
# Allow this environment to be available on the LAUNCHERS (+) sign on the left
!./myfancyenv/bin/python -m ipykernel install --user --name fancy --display-name "My Python"
# Allow this environment to be available on the LAUNCHERS (+) sign on the left
!./myfancyenv/bin/python -m ipykernel install --user --name fancy --display-name "My Python"
Installed kernelspec fancy in /jetfs/home/mblaschek/.local/share/jupyter/kernels/fancy
Wait a bit and the new python environment should be available on your launchers.
In [10]:
Copied!
!jupyter kernelspec list
!jupyter kernelspec list
Available kernels: fancy /jetfs/home/mblaschek/.local/share/jupyter/kernels/fancy myenv /jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv python3 /jetfs/spack/opt/spack/linux-rhel8-skylake_avx512/gcc-8.3.1/miniconda3-4.8.2-3m7b6t2kgedyr3jnd2nasmgiq7wm27iv/share/jupyter/kernels/python3
Add PATH to your user environment¶
In order to get full access to all tools installed in your local environment, add it to your PATH.
Open your kernel.json file located /jetfs/home/mblaschek/.local/share/jupyter/kernels/myenv/kernel.json:
{
"argv": [
"/jetfs/home/mblaschek/.conda/envs/myenv/bin/python",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
],
"env" : {
"PATH": "/jetfs/home/mblaschek/.conda/envs/myenv/bin/:${PATH}",
"LD_LIBRARY_PATH": "/jetfs/home/mblaschek/.conda/envs/myenv/lib:${LD_LIBRARY_PATH}"
},
"display_name": "MYENV",
"language": "python"
}