Connecting to remote IPython Kernels¶

In order to connect a few things need to be setup:

1. install Jupyter Console
2. install Remote Ikernel
3. configure the kernel

These commands will install the required packages into your ~/.local/bin and ~/.local/lib directories by default in Linux/Mac. If you are using the anaconda then the packages will be installed there.

Make sure that the /bin path is in your $PATH environment variable.

In [2]:

# This shows you which python distribution you are using
# pip is the python installer / not conda
!which pip3

# This shows you which python distribution you are using # pip is the python installer / not conda !which pip3

/home/opt/spack/opt/spack/linux-centos6-sandybridge/gcc-5.3.0/anaconda3-2020.07-6p2eqnrd2mu4masynsdqvrbfdit7fyuk/bin/pip3

Install the required packages¶

In [10]:

!pip3 -q install jupyter-console

!pip3 -q install jupyter-console

In [2]:

!pip3 -q install remote_ikernel

!pip3 -q install remote_ikernel

In [17]:

# this might not be necessary for everyone
# adjust the path accordingly to make sure that jupyter-kernelspec, jupyter and remote_ikernel can be found.
%env PATH=/home/${USER}/.local/bin:$PATH

# this might not be necessary for everyone # adjust the path accordingly to make sure that jupyter-kernelspec, jupyter and remote_ikernel can be found. %env PATH=/home/${USER}/.local/bin:$PATH

env: PATH=/home/mblaschek/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Configure remote connection¶

Depending on your install, you might have one or two selections here. jupyter-kernelspec shows you the kernels that you can use. The one you installed the above packages with should be here. We don't need these anyway. Just so you know about that.

jupyter-kernelspec [cmd]
    list
        List installed kernel specifications.
    install
        Install a kernel specification directory.
    remove
        Remove one or more Jupyter kernelspecs by name.

In [18]:

!jupyter-kernelspec list

!jupyter-kernelspec list

Available kernels:
  anaconda3    /home/mblaschek/.local/share/jupyter/kernels/anaconda3
  python3      /home/mblaschek/.local/share/jupyter/kernels/python3

In [19]:

# This is the relevant file.
# It contains the path to the executable and some names
!cat /home/${USER}/.local/share/jupyter/kernels/anaconda3/kernel.json

# This is the relevant file. # It contains the path to the executable and some names !cat /home/${USER}/.local/share/jupyter/kernels/anaconda3/kernel.json

{
 "argv": [
  "/home/mblaschek/anaconda3/bin/python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python 3(Ana)",
 "language": "python"
}

Adding a new kernel¶

on SRVX1, SRVX2, SRVX8 it is possible to launch a script called kernel-launcher.

kernel-launcher [module] [executable] [options]

typically we launch this with python processes such as

kernel-launcher miniconda3 ipython 

kernel-launcher will load the module and execute the executable (here: ipython)

Modules available on SRVX1¶

>>> module av

--- /home/opt/spack/share/spack/modules/linux-centos6-sandybridge ---
anaconda3/2020.07-gcc-5.3.0     git/2.29.0-gcc-5.3.0        netcdf-fortran/4.5.3-gcc-5.3.0  
eccodes/2.18.0-gcc-5.3.0        hdf5/1.10.7-gcc-5.3.0       openmpi/3.1.6-gcc-5.3.0         
enstools/2020.11.dev-gcc-5.3.0  miniconda3/4.8.2-gcc-5.3.0  proj/7.1.0-gcc-5.3.0            
gcc/5.3.0-gcc-5.3.0             netcdf-c/4.7.4-gcc-5.3.0    zlib/1.2.11-gcc-5.3.0

Modules available on SRVX8¶

>>> module av

--- /opt/spack/share/spack/modules/linux-centos6-haswell/ ---
anaconda3/2020.07-gcc-5.3.0    hdf5/1.10.7-gcc-5.3.0          openmpi/3.1.6-gcc-5.3.0
eccodes/2.18.0-gcc-5.3.0       miniconda3/4.8.2-gcc-5.3.0     proj/7.1.0-gcc-5.3.0
gcc/4.4.7-gcc-5.3.0            netcdf-c/4.7.4-gcc-5.3.0       python/3.8.5-gcc-5.3.0
gcc/5.3.0-gcc-5.3.0            netcdf-fortran/4.5.3-gcc-5.3.0 zlib/1.2.11-gcc-5.3.0

In order to use these different modules one has to specify them in the call:

1. Module: miniconda3
2. Name: PyS1 for Python on srvx1
3. Host: mblaschek@srvx1.img.univie.ac.at username and server

In order to allow easy access adding the public key to ssh authentication is recommended. Otherwise set export SSH_ASKPASS=1 and you should be prompted each time the connection is established.

In [20]:

!remote_ikernel manage \
    --add \
    --kernel_cmd='kernel-launcher miniconda3 ipython kernel -f {connection_file}' \
    --name='PyS1' \
    --interface=ssh \
    --host=mblaschek@srvx1.img.univie.ac.at

!remote_ikernel manage \ --add \ --kernel_cmd='kernel-launcher miniconda3 ipython kernel -f {connection_file}' \ --name='PyS1' \ --interface=ssh \ --host=mblaschek@srvx1.img.univie.ac.at

Added kernel ['rik_ssh_mblaschek_srvx1_img_univie_ac_at_pys1']: SSH mblaschek@srvx1.img.univie.ac.at PyS1.

In [21]:

# Here we see the newly added kernel
!jupyter-kernelspec list

# Here we see the newly added kernel !jupyter-kernelspec list

Available kernels:
  anaconda3                                        /home/mblaschek/.local/share/jupyter/kernels/anaconda3
  python3                                          /home/mblaschek/.local/share/jupyter/kernels/python3
  rik_ssh_mblaschek_srvx1_img_univie_ac_at_pys1    /home/mblaschek/.local/share/jupyter/kernels/rik_ssh_mblaschek_srvx1_img_univie_ac_at_pys1

In [23]:

# Here we can see that the remote_ikernel actually does.
!cat  /home/mblaschek/.local/share/jupyter/kernels/rik_ssh_mblaschek_srvx1_img_univie_ac_at_pys1/kernel.json

# Here we can see that the remote_ikernel actually does. !cat /home/mblaschek/.local/share/jupyter/kernels/rik_ssh_mblaschek_srvx1_img_univie_ac_at_pys1/kernel.json

{
  "argv": [
    "/usr/bin/python3",
    "-m",
    "remote_ikernel",
    "--interface",
    "ssh",
    "--host",
    "mblaschek@srvx1.img.univie.ac.at",
    "--kernel_cmd",
    "kernel-launcher miniconda3 ipython kernel -f {host_connection_file}",
    "{connection_file}"
  ],
  "display_name": "SSH mblaschek@srvx1.img.univie.ac.at PyS1",
  "remote_ikernel_argv": [
    "/home/mblaschek/.local/bin/remote_ikernel",
    "manage",
    "--add",
    "--kernel_cmd=kernel-launcher miniconda3 ipython kernel -f {connection_file}",
    "--name=PyS1",
    "--interface=ssh",
    "--host=mblaschek@srvx1.img.univie.ac.at"
  ]
}

In [ ]:

# this will launch the anaconda3 kernel from the list
!jupyter console --existing=anaconda3

# this will launch the anaconda3 kernel from the list !jupyter console --existing=anaconda3

In [ ]:

# This will launch the remote kernel in a console
!jupyter console --existing=rik_ssh_mblaschek_srvx1_img_univie_ac_at_pys1

# This will launch the remote kernel in a console !jupyter console --existing=rik_ssh_mblaschek_srvx1_img_univie_ac_at_pys1

of course the remote kernel can be used in a local jupyter notebook or lab.

Adding a custom kernel¶

The only thing that you need is the ipython executable and you can add any kernel to your lab.

In [ ]:

!remote_ikernel manage \
    --add \
    --kernel_cmd='/opt/anaconda3/bin/ipython kernel -f {connection_file}' \
    --name='MyP3' \
    --interface=ssh \
    --host=mblaschek@srvx1.img.univie.ac.at

!remote_ikernel manage \ --add \ --kernel_cmd='/opt/anaconda3/bin/ipython kernel -f {connection_file}' \ --name='MyP3' \ --interface=ssh \ --host=mblaschek@srvx1.img.univie.ac.at

Remarks¶

When running kernels on remote machines, the notebooks themselves will be saved onto the local filesystem, but the kernel will only have access to filesystem of the remote machine running the kernel. If you need shared directories, set up sshfs between your machines.

In [ ]:

---

Last update: January 28, 2022
Created: October 14, 2021