/!\ IMPORTANT: this is a set of hands-on tutorials: participants are expected to bring a laptop and pre-install software in advance to make the best use of time during the proposed tutorials. If for some reason you are unable to fulfill this pre-requisite, try to seat close to an attendee that is able to perform these tasks.

Note: in the following instructions, terminal commands are prefixed by a virtual prompt $>which obviously does not belong to the command.

Online accounts

Kindly create in advance the various accounts for the cloud services we might use, i.e.:

UL HPC account

You need to have an account on our platform. See https://hpc-docs.uni.lu/accounts/

Software List

The following software should be installed, depending on your running platform. Detailed instructions for each OS are depicted below.

Platform Software Description Usage
Mac OS Homebrew The missing package manager for macOS brew install ...
Mac OS iTerm2 enhanced Terminal
Windows Chocolatey Package Manager for Windows choco install ...
Windows Windows Subsystem for Linux (WSL) Emulation-like translation of Linux kernel system calls
Windows Ubuntu over WSL Linux Ubuntu on Windows (recommended)
Windows Windows Terminal
Windows MobaXTERM Terminal with tabbed SSH client
Windows SourceTree (optional) enhanced git GUI
Windows/Linux Virtual Box Free hypervisor provider for Vagrant
Windows/Linux Vagrant Reproducible environments made easy.
Linux Docker for Ubuntu Lightweight Reproducible Containers
Windows Docker for Windows Lightweight Reproducible Containers

Follow the below custom instructions depending on your running platform and Operating System.

Microsoft Windows

Chocolatey: The Package Manager for Windows

Follow Installation instructions on https://chocolatey.org/ - install it as an administrator PowerShell. You'll probably need to reboot your laptop.

Chocolatey Installation

With PowerShell, you must ensure Get-ExecutionPolicy is not Restricted.

  • Right click on the Windows starting boutton and choose Windows PowerShell

  • Run these three commands

Get-ExecutionPolicy
### if it returns  Restricted then go to the next step
Set-ExecutionPolicy AllSigned   ## or Set-ExecutionPolicy Bypass -Scope Process
Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))
  • Then use chocolatey to install a software
choco.exe install virtualbox

WSL and Ubuntu over Windows

The Windows Subsystem for Linux makes this all possible by providing emulation-like translation of Linux kernel system calls to the Windows kernel. So when a Linux app like Bash executes and makes system calls, the Windows Subsystem for Linux translates those calls into Windows system calls and Bash executes just like on Linux.

WSL Installation

In the Start Menu: select "Turn Windows features on or off"

  • Tick "Windows Subsystem for Linux"
  • reboot to complete the install

Then Enable the Developer mode

  • In Start Menu: select "Developers Settings"
  • Turn on Developer Mode

Now you can Install Ubuntu within the Microsoft Store

  • Search for Ubuntu and install it
    • create a new UNIX username (prefer to follow the guidelines on ULHPC facility: first letter of your firstname, followed by your lastname, all in lower case. Ex for John Doe: jdoe)
    • add the new password
    • Your windows system drive are exposed in the /mnt/ directory
      • Ex: C:\Users\MyLogin\Downloads\File.txt will be located under /mnt/c/Users/MyLogin/Downloads/File.txt

BEWARE that your Linux file system is stored in a hidden folder, under %userprofile%\AppData\Local\Packages. In particular for Ubuntu, the files are located under the CanonicalGroupLimited.UbuntuonWindows_<hash>\LocalState\rootfs folder. Note that you're NOT supposed to tamper these files. But, if you need to view or back up some files, you'll find them stored in a hidden folder.

Ubuntu Image Customization

In the Ubuntu bash, you may want to install of-my-zsh and the Powerlevel10k prompt following the following guide:

You will need to enable by default the good font (top left window icon / Properties / Fonts)

Microsoft Terminal

You probably want to install then Windows Terminal It offers the ability to use multiple shell environment in one terminal.

Changing the Powershell prompt

Install posh-git and oh-my-posh:

Install-Module posh-git -Scope CurrentUser
Install-Module oh-my-posh -Scope CurrentUser
Install-Module -Name PSReadLine -Scoope CurrentUser -Force -SkipPublisherCheck 
Set-PoshPrompt -Theme Agnoster

If you get an error message "Running scripts is disabled on this system", you have to change the PowerShell execution policy which doesn't allow to run scripts:

Get-ExecutionPolicy -List
Set-ExecutionPolicy RemoteSigned -Scope CurrentUser

To make these changes permanent, append the following lines in the Powershell profile (run notepad $PROFILE):

Import-Module posh-git
Import-Module oh-my-posh
Set-PoshPrompt -Theme Agnoster

You will also need to enable the Cascadia Code PL font by adding into the Windows Terminal Parameters the following lines in the settings.json files under the default

"defaults":
{
    "font": 
            {
                "face": "MesloLGM NF"
            }
        },
        "list": 
        [
            {
                "commandline": "powershell.exe",
                "guid": "{61c54bbd-c2c6-5271-96e7-009a87ff44bf}",
                "hidden": false,
                "name": "Windows PowerShell"
                "fontFace": "MesloLGM NF",
            },
        ]
},

MobaXterm to Run Graphical Linux Apps

Now you can install MobaXterm using chocolatey (within an administrator Powershell -- You may need to enable the appropriate powerline font by defaults in the Properties of the Administrator Powershell):

$ choco.exe install mobaxterm          # Enhanced X11 Terminal for Windows
$ choco.exe install vcxsrv

Alternatively, you can consider using VcXsrv as an X-server yet our training sessions will assume you rely on MobaXterm. However if you wish to install VcXsrv to natively run X application from the windows terminal, proceed as follows:

  • Run XLaunch Wizard
  • Accept the default options BUT pay attention to Save the configuration file at the last step in %appdata%\Microsoft\Windows\Start Menu\Programs\Startup (eventually search for the 'Start Menu' directory in the explorer).
    • DENY (cancel) the Firewall request

Every time you run MobaXterm, make sure that the X server is running - check it by clicking on the associate button. Select the WSL/Ubuntu profile. We need to configure WSL to send the display of its graphical apps over to Windows. Failing to do so will result in the fact that graphical apps will attempt to load natively inside of WSL and nothing will show up.

To do that, we need to set the DISPLAY environment variable within WSL. Run nano ~/.profile to append the following lines:

# Check WSL version with 'wsl.exe -l -v'
# If using WSL 1:
export DISPLAY=:0
# If using WSL 2:
export DISPLAY="$(/sbin/ip route | awk '/default/ { print $3 }'):0"

IDE / Programming Editor

You probably want to install the following editors/IDE:

Useful applications install

Then, while most of the below software are covered in the trainings, if you want a fast setup, once you have Chocolatey installed, run the following within an administrator Powershell:

$ choco.exe install git gitflow-avh    # (newer) Git stuff
$ choco.exe install mobaxterm          # Enhanced X11 Terminal for Windows
$ choco.exe install virtualbox         # install virtualbox -- see https://www.virtualbox.org/
$ choco.exe install vagrant            # install Vagrant    -- see https://www.vagrantup.com/downloads.html
$ choco.exe install docker-desktop     # install Docker -- https://docs.docker.com/engine/installation/mac/

Update to WSL 2

Better performances of your Linux subsystem can be obtained by migrating to WSL 2.

In an administrator Powershell, enable the Virtual Machine Platform

Enable-WindowsOptionalFeature -Online -FeatureName VirtualMachinePlatform

Then you can get the list of your WSL systems and convert them as follows:

$ wsl -l -v
  NAME     STATE     VERSION
* Ubuntu   Stopped   1
# Convert -- Adapt the Distribution name accordinaly
$ wsl --set-version Ubuntu 2

For setting all future distributions to use WSL 2, you will need to use the following command:

$ wsl --set-default-version 2

Finally, verify that your changes worked:

$ wsl -l -v    # OR wsl --list --verbose
  NAME     STATE     VERSION
* Ubuntu   Stopped   2

Mac OS X

Install iterm2 and Homebrew Once you have Homebrew installed:

$ brew install git git-flow    # (newer) Git stuff
$ brew install mkdocs               # (optional) install mkdocs
$ brew install pyenv pyenv-virtualenv direnv # see https://varrette.gforge.uni.lu/tutorials/pyenv.html
$ brew install virtualbox      # install virtualbox -- see https://www.virtualbox.org/
$ brew install vagrant         # install Vagrant    -- see https://www.vagrantup.com/downloads.html
$ brew install vagrant-manager # see http://vagrantmanager.com/
$ brew install docker          # install Docker -- https://docs.docker.com/engine/installation/mac/
# Note that you probably want to install Firefox, Chrome etc. with brew
$ brew install firefox
$ brew install google-chrome

Note: later on, you might wish to use the following shell function to update the software installed using Homebrew.

bup () {
    echo "Updating your [Homebrew] system"
    brew update
    brew upgrade
    brew cu
    brew cleanup
    brew cask cleanup
}

Linux (Debian / Ubuntu)

# Adapt the package names (and package manager) in case you are using another Linux distribution.
$ sudo apt-get update
$ sudo apt-get install git git-flow build-essential
$ sudo apt-get install rubygems virtualbox vagrant virtualbox-dkms

For Docker, choose your distribution from https://docs.docker.com/engine/installation/linux/ and follow the instructions. You need a reasonably new kernel version (3.10 or higher). Here are detailed instuctions per OS:

You may want to install Sublime Text:

wget -qO - https://download.sublimetext.com/sublimehq-pub.gpg | sudo apt-key add -
sudo apt-get install apt-transport-https
echo "deb https://download.sublimetext.com/ apt/stable/" | sudo tee /etc/apt/sources.list.d/sublime-text.list
sudo apt-get update
sudo apt-get install sublime-text

Check out this Sublime Text 3 package configuration:

git clone https://github.com/nickjj/sublime-text-3-packages.git ~/.config/sublime-text-3

Then test the configuration with subl .

Post-Installations checks

Git:

(Eventually) Make yourself known to Git

$ git config –-global user.name  "Firstname LastName"              # Adapt accordingly
$ git config –-global user.email "Firstname.LastName@domain.org"   # Adapt with your mail
# Eventually, if you have a GPG key, use the public key to sign your commits/tags
$ git config --global user.helper osxkeychain       # Only on Mac OS
$ git config --global user.signingkey <fingerprint> # Ex: git config --global user.signingkey 5D08BCDD4F156AD7
# you can get your key fingerprint (prefixed by 0x) with 'gpg -K --fingerprint | grep sec'

To clone and install this repository, follow the installation instructions.

Vagrant

Ensure that vagrant is running and has the appropriate plugins from the command line

$ vagrant --version
Vagrant 2.2.13

Docker (only required for containers tutorials)

Launch the Docker app and then check that the Docker works:

$ docker info
Containers: 9
 Running: 0
 Paused: 0
 Stopped: 9
Images: 12
Server Version: 18.03.1-ce
[...]
  • Pull the docker containers we might need for the concerned tutorial
$ docker pull centos
  • Login onto you Docker hub account (take note of your Docker Hub ID and password).
    • With docker installed, run
$ docker login -u <your docker hub ID>

and enter your password.

Note that if the Docker installation fails, you can use http://play-with-docker.com/ to try Docker, but it won't work if all of us try it once! So use it only as a last resort, and it is up to you to use any important information (like the Docker hub account) inside it.

Mkdocs

It probably makes sense to install mkdocs to be able to generate locally the current documentation.

Follow for that the instructions provided on the ../rtfd.md.