By ULHPC Licence GitHub issues Github Documentation Status GitHub forks

UL HPC Tutorial: Use Jupyter notebook on UL HPC

  Copyright (c) 2020  UL HPC Team <hpc-team@uni.lu>

Authors: Clément Parisot (updated by Frédéric Pinel)

Python is a high-level interpreted language widely used in research. It lets you work quickly and comes with a lot of available packages which give more useful functionalities.

Reserve a node

si
# or si-gpu

Load Python and install required modules in a virtualenv

If you have never used virtualenv before, please have a look at Python1 tutorial.

# Load your prefered version of Python

# Optional: you can choose the soon-to-be released 2019b software set:
unset MODULEPATH
# For broadwell processor nodes:
module use /opt/apps/resif/iris/2019b/broadwell/modules/all
# OR for skylake processor nodes:
module use /opt/apps/resif/iris/2019b/skylake/modules/all
# OR for skylake/GPU nodes:
module use /opt/apps/resif/iris/2019b/gpu/modules/all
# ELSE, keep the current software set (nothing required, but actual python versions vary)

module load lang/Python/3.7.2-GCCcore-8.2.0   # optionally: 2019b skylake, lang/Python/3.7.4-GCCcore-8.3.0
# Create a new virtualenv
git clone https://github.com/ULHPC/tutorials
cd tutorials/python/advanced/jupyter/
pip install --user virtualenv
PATH=$PATH:$HOME/.local/bin
virtualenv venv
source venv/bin/activate

Now we need to install all the modules needed. They are listed in the requirements.txt at the root of our tutorial directory. Here is the list of essentials ones:

# jupyter himself
pip install jupyter
# matplotlib to plot the graph inside your notebook
pip install matplotlib
# To use our virtualenv in the notebook, we need to install this module
pip install ipykernel

To save the installed packages:

pip freeze -l > requirements.txt

To install everything:

pip install -r requirements.txt

Now everything is installed properly.

Create your own kernel and launch your Notebook

In order to access to all the modules we have installed inside your Notebook, we will need to create a new Kernel and use it inside Jupyter.

To do so, let's use ipykernel.

python -m ipykernel install --user --name=venv # The "venv" name here is to give your kernel a name and it will install it in your $HOME path. If a similarly named kernel exists, it will be overwritten.
# In case you would like your kernel to be installed into your active conda environment (<YOURCONDAPATH-PREFIX>/miniconda3/envs/jupyter_env/share/jupyter/kernels/), use the command below. This may be preferred as it encapsulates everything into a single environment, but would deviate from the virtualenv-based configuration above more than necessary for this tutorial.
#python -m ipykernel install --sys-prefix --name 'mylocalkernel'
# A completely custom specification of the path is *discouraged* as the resulting warning about the path not being in the "default" places and might hence not be found is very real. This means that the kernel can not be selected from the "New" dialog in the Jupyter interface. S. a. https://scipy-ipython.readthedocs.io/en/latest/install/kernel_install.html#kernels-for-different-environments for further information.
# Use only if you know what you do!!!
#python -m ipykernel install --prefix=./ --name 'myhyperlocalkernel'

Now everything is installed properly using conda.

Now we will have to start our first notebook. To have access to it from the outside, we will need to run it on the correct IP of the node. This simple command permits to start a new notebook with the correct IP. Please ensure that you are running the command inside the correct directory!

The --no-browser command is used to disable the openning of the browser after the start of the notebook. We use --generate-config at first to generate a default configuration. The default configuration is stored in ~/.jupyter/jupyter_notebook_config.py

To make things easier, we will protect our Notebook with a password. You have just to choose a password after typing the jupyter notebook password command. A hash of your password will be stored in the jupyter config file.

#cd tutorials/python/advanced/jupyter # Only needed if you do not follow from above
jupyter notebook --generate-config
jupyter notebook password
jupyter notebook --ip $(ip addr show em1 | grep 'inet ' | awk '{print $2}' | cut -d/ -f1) --no-browser

At the end of the command, you should see a link like this:

[I 17:45:05.756 NotebookApp] Serving notebooks from local directory: /mnt/irisgpfs/users/cparisot/Python2
[I 17:45:05.756 NotebookApp] 0 active kernels
[I 17:45:05.756 NotebookApp] The Jupyter Notebook is running at:
[I 17:45:05.757 NotebookApp] http://172.17.6.55:8888/
[I 17:45:05.757 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

Now we need to create an SSH tunneling in order to access this URL on your laptop. We will forward every local requests made on your localhost address of your laptop the cluster. The simpliest way to do so is to type this command if your are on Linux:

ssh -NL 8888:<IP OF YOUR NODE>:8888 iris-cluster
# In my example, my command will be: ssh -NL 8888:172.17.6.55:8888 iris-cluster

Then, by accessing to your local port 8888 on the localhost address 127.0.0.1, you should see the Python notebook.

http://127.0.0.1:8888/.

If you haven't chosen a password to protect your notebook, please append the token to the URL before accessing your Notebook.

Run our first notebook

  • Just click on the Notebook jupyter/Monte-Carlo calculation of pi.ipynb.
  • Change the kernel for the venv one
  • Go onto Kernel tab
  • Choose Change kernel
  • Select our previously generated kernel called venv
  • Try to run the code of the notebook in the kernel by using 'Alt-Enter' keys.