Connecting to remote IPython Kernels¶
In order to connect a few things need to be setup:
- install Jupyter Console
- install Remote Ikernel
- configure the kernel
These commands will install the required packages into your ~/.local/bin and ~/.local/lib directories by default in Linux/Mac.
If you are using the anaconda then the packages will be installed there.
Make sure that the /bin path is in your $PATH environment variable.
# This shows you which python distribution you are using
# pip is the python installer / not conda
!which pip3
/home/opt/spack/opt/spack/linux-centos6-sandybridge/gcc-5.3.0/anaconda3-2020.07-6p2eqnrd2mu4masynsdqvrbfdit7fyuk/bin/pip3
Install the required packages¶
!pip3 -q install jupyter-console
!pip3 -q install remote_ikernel
# this might not be necessary for everyone
# adjust the path accordingly to make sure that jupyter-kernelspec, jupyter and remote_ikernel can be found.
%env PATH=/home/${USER}/.local/bin:$PATH
env: PATH=/home/mblaschek/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Configure remote connection¶
Depending on your install, you might have one or two selections here. jupyter-kernelspec shows you the kernels that you can use. The one you installed the above packages with should be here. We don't need these anyway. Just so you know about that.
jupyter-kernelspec [cmd]
list
List installed kernel specifications.
install
Install a kernel specification directory.
remove
Remove one or more Jupyter kernelspecs by name.!jupyter-kernelspec list
Available kernels: anaconda3 /home/mblaschek/.local/share/jupyter/kernels/anaconda3 python3 /home/mblaschek/.local/share/jupyter/kernels/python3
# This is the relevant file.
# It contains the path to the executable and some names
!cat /home/${USER}/.local/share/jupyter/kernels/anaconda3/kernel.json
{
"argv": [
"/home/mblaschek/anaconda3/bin/python",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
],
"display_name": "Python 3(Ana)",
"language": "python"
}
Adding a new kernel¶
on SRVX1, SRVX2, SRVX8 it is possible to launch a script called kernel-launcher.
kernel-launcher [module] [executable] [options]typically we launch this with python processes such as
kernel-launcher miniconda3 ipython kernel-launcher will load the module and execute the executable (here: ipython)
Modules available on SRVX1¶
>>> module av
--- /home/opt/spack/share/spack/modules/linux-centos6-sandybridge ---
anaconda3/2020.07-gcc-5.3.0 git/2.29.0-gcc-5.3.0 netcdf-fortran/4.5.3-gcc-5.3.0
eccodes/2.18.0-gcc-5.3.0 hdf5/1.10.7-gcc-5.3.0 openmpi/3.1.6-gcc-5.3.0
enstools/2020.11.dev-gcc-5.3.0 miniconda3/4.8.2-gcc-5.3.0 proj/7.1.0-gcc-5.3.0
gcc/5.3.0-gcc-5.3.0 netcdf-c/4.7.4-gcc-5.3.0 zlib/1.2.11-gcc-5.3.0
Modules available on SRVX8¶
>>> module av
--- /opt/spack/share/spack/modules/linux-centos6-haswell/ ---
anaconda3/2020.07-gcc-5.3.0 hdf5/1.10.7-gcc-5.3.0 openmpi/3.1.6-gcc-5.3.0
eccodes/2.18.0-gcc-5.3.0 miniconda3/4.8.2-gcc-5.3.0 proj/7.1.0-gcc-5.3.0
gcc/4.4.7-gcc-5.3.0 netcdf-c/4.7.4-gcc-5.3.0 python/3.8.5-gcc-5.3.0
gcc/5.3.0-gcc-5.3.0 netcdf-fortran/4.5.3-gcc-5.3.0 zlib/1.2.11-gcc-5.3.0
In order to use these different modules one has to specify them in the call:
- Module:
miniconda3 - Name:
PyS1for Python on srvx1 - Host:
mblaschek@srvx1.img.univie.ac.atusername and server
In order to allow easy access adding the public key to ssh authentication is recommended. Otherwise set export SSH_ASKPASS=1 and you should be prompted each time the connection is established.
!remote_ikernel manage \
--add \
--kernel_cmd='kernel-launcher miniconda3 ipython kernel -f {connection_file}' \
--name='PyS1' \
--interface=ssh \
--host=mblaschek@srvx1.img.univie.ac.at
Added kernel ['rik_ssh_mblaschek_srvx1_img_univie_ac_at_pys1']: SSH mblaschek@srvx1.img.univie.ac.at PyS1.
# Here we see the newly added kernel
!jupyter-kernelspec list
Available kernels: anaconda3 /home/mblaschek/.local/share/jupyter/kernels/anaconda3 python3 /home/mblaschek/.local/share/jupyter/kernels/python3 rik_ssh_mblaschek_srvx1_img_univie_ac_at_pys1 /home/mblaschek/.local/share/jupyter/kernels/rik_ssh_mblaschek_srvx1_img_univie_ac_at_pys1
# Here we can see that the remote_ikernel actually does.
!cat /home/mblaschek/.local/share/jupyter/kernels/rik_ssh_mblaschek_srvx1_img_univie_ac_at_pys1/kernel.json
{
"argv": [
"/usr/bin/python3",
"-m",
"remote_ikernel",
"--interface",
"ssh",
"--host",
"mblaschek@srvx1.img.univie.ac.at",
"--kernel_cmd",
"kernel-launcher miniconda3 ipython kernel -f {host_connection_file}",
"{connection_file}"
],
"display_name": "SSH mblaschek@srvx1.img.univie.ac.at PyS1",
"remote_ikernel_argv": [
"/home/mblaschek/.local/bin/remote_ikernel",
"manage",
"--add",
"--kernel_cmd=kernel-launcher miniconda3 ipython kernel -f {connection_file}",
"--name=PyS1",
"--interface=ssh",
"--host=mblaschek@srvx1.img.univie.ac.at"
]
}
# this will launch the anaconda3 kernel from the list
!jupyter console --existing=anaconda3
# This will launch the remote kernel in a console
!jupyter console --existing=rik_ssh_mblaschek_srvx1_img_univie_ac_at_pys1
of course the remote kernel can be used in a local jupyter notebook or lab.
Adding a custom kernel¶
The only thing that you need is the ipython executable and you can add any kernel to your lab.
!remote_ikernel manage \
--add \
--kernel_cmd='/opt/anaconda3/bin/ipython kernel -f {connection_file}' \
--name='MyP3' \
--interface=ssh \
--host=mblaschek@srvx1.img.univie.ac.at
Remarks¶
When running kernels on remote machines, the notebooks themselves will be saved onto the local filesystem, but the kernel will only have access to filesystem of the remote machine running the kernel. If you need shared directories, set up sshfs between your machines.
Created: January 26, 2023