HPC Management of Sequential and Embarrassingly Parallel Jobs
Copyright (c) 2020 S. Varrette and UL HPC Team <hpc-team@uni.lu>
For many users, the reason to consider (or being encouraged) to offload their computing executions on a (remote) HPC or Cloud facility is tied to the limits reached by their computing devices (laptop or workstation). It is generally motivated by time constraints:
"My computations take several hours/days to complete. On an HPC, it will last a few minutes, no?"
or search-space explorations:
"I need to check my application against a huge number of input pieces (files) - it worked on a few of them locally but takes ages for a single check. How to proceed on HPC?"
For a large majority of you, these questions arise when executing programs that you traditionally run on your laptop or workstation and which consists in:
- a (well-known) application installed on your system, iterated over multiple input conditions configured by specific command-line arguments / configuration files
- a compiled program (C, C++, Java, Go etc.), iterated over multiple input conditions
- your favorite R or python (custom) development scripts, iterated again over multiple input conditions
Be aware that in most of the cases, these applications are inherently SERIAL: These are able to use only one core when executed. You thus deal with what is often call a Bag of (independent) tasks, also referred to as embarrassingly parallel tasks. The objective of this practical session is to guide you toward the effective management of such tasks in a way that optimize both the resource allocation and the completion time.
Pre-requisites
Ensure you are able to connect to the UL HPC clusters.
In particular, recall that the module command is not available on the access frontends.
If you have never configured GNU Screen before, and while not strictly mandatory, we advise you to rely on our customized configuration file for screen .screenrc available on Github and available on the access nodes under /etc/dotfiles.d/screen/.screenrc
### Access to ULHPC cluster - here iris
(laptop)$> ssh iris-cluster
# /!\ Advanced (but recommended) best-practice:
# always work within an GNU Screen session named with 'screen -S <topic>' (Adapt accordingly)
# IIF not yet done, copy ULHPC .screenrc in your home
(access)$> cp /etc/dotfiles.d/screen/.screenrc ~/
Now you'll need to pull the latest changes in your working copy of the ULHPC/tutorials you should have cloned in ~/git/github.com/ULHPC/tutorials (see "preliminaries" tutorial)
(access)$> cd ~/git/github.com/ULHPC/tutorials
(access)$> git pull
Now configure a dedicated directory ~/tutorials/sequential for this session
# return to your home
(access)$> mkdir -p ~/tutorials/sequential
(access)$> cd ~/tutorials/sequential
# create a symbolic link to the reference material
(access)$> ln -s ~/git/github.com/ULHPC/tutorials/sequential/basics ref.d
Advanced users (eventually yet strongly recommended), create a GNU Screen session you can recover later - see "Getting Started" tutorial
# /!\ Advanced (but recommended) best-practice:
# Always work within an GNU Screen session named with 'screen -S <topic>' (Adapt accordingly)
# _note_: assumes you have previously copied ULHPC .screenrc in your homedir
(access-iris)$> screen -S HPC-school
# CTRL + a c: (create) creates a new Screen window. The default Screen number is zero.
# CTRL + a n: (next) switches to the next window.
# CTRL + a p: (prev) switches to the previous window.
# CTRL + a A: (title) rename the current window
# CTRL + a d: (detach) detaches from a Screen -
# Once detached:
# screen -ls : list available screen
# screen -x reattach to a past screen
A sample "Stress Me!" parameter exploration
In this sample scenario, we will mimic the exploration of integer parameters within a continuous range [1..n] for serial (1-core) tasks lasting different amount time (here ranging ranging from 1 to n seconds). This is very typical: your serial application is likely to take more or less time depending on its execution context (typically controlled by command line arguments).
To keep it simple, we will assume the n=30 by default.
You will use a script, scripts/run_stressme, to reproduce a configurable amount of stress on the system set for a certain amount of time (by default: 20s) passed as parameter using the 'stress' command.
We would like thus to conduct the following executions:
run_stressme 1 # execution time: 1s
run_stressme 2 # execution time: 2s
run_stressme 3 # execution time: 3s
run_stressme 4 # execution time: 4s
run_stressme 5 # execution time: 5s
run_stressme 6 # execution time: 6s
run_stressme 7 # execution time: 7s
[...]
run_stressme 27 # execution time: 27s
run_stressme 28 # execution time: 28s
run_stressme 29 # execution time: 29s
run_stressme 30 # execution time: 30s
Running this job campaign sequentially (one after the other on the same core) would take approximatively 465s.
The below table wrap up the sequential times required for the completion of the job campaign depending on the number of run_stressme tasks, and the theoretical optimal time corresponding to the slowest task to complete when using an infinite number of computing resources.
| n = Number of tasks | Expected Seq. time to complete | Optimal time |
|---|---|---|
| 1 | 1s | 1s |
| 10 | 55s (~1 min) | 10s |
| 30 (default) | 465s (7 min 45s) | 30s |
| 100 | 5050s (1h 24 min 10s) | 100s |
The objective is of course to optimize the time required to conclude this exploration
Single task run (interactive)
Let's first test this command in an interactive jobs:
### Access to ULHPC cluster (if not yet done)
(laptop)$> ssh iris-cluster
### Have an interactive job
# ... either directly
(access)$> si
# ... or using the HPC School reservation 'hpcschool'if needed - use 'sinfo -T' to check if active and its name
# (access)$> si --reservation=hpcschool
(node)$>
In another terminal (or another screen tab/windows), connect to that job and run htop
# check your running job
(access)$> sq
# squeue -u $(whoami)
JOBID PARTIT QOS NAME USER NODE CPUS ST TIME TIME_LEFT PRIORITY NODELIST(REASON)
2171206 [...]
# Connect to your running job, identified by its Job ID
(access)$> sjoin 2171206 # /!\ ADAPT job ID accordingly, use <TAB> to have it autocatically completed
(node)$> htop # view of all processes
# F5: tree view
# u <name>: filter by process of <name>
# q: quit
Note that you have only one core reserved by default in a interactive job: you may thus see activities on the other cores via htop, they are corresponding to processes run by the other users. Note that you can use the --exclusive flag upon reservation to request en exclusive access to a full node.
Now execute the run_stressme command in the first terminal/windows to see its effect on the processor load:
# Run directly
(node)$> ~/git/github.com/ULHPC/tutorials/sequential/basics/scripts/run_stressme
# /usr/bin/stress -c 1 -t 20
stress: info: [59918] dispatching hogs: 1 cpu, 0 io, 0 vm, 0 hdd
stress: info: [59918] successful run completed in 20s
# Now let's create a convenient path to the scripts under the dedicated directory
# used for this session
(node)$> cd ~/tutorials/sequential
# create a symbolic link to the script directory
(node)$> ln -s ref.d/scripts . # <-- don't forget the trailing '.' means 'here'
(node)$> ls scripts/run_stressme # should not fail
scripts/run_stressme
(node)$> ./scripts/run_stressme
stress: info: [59918] dispatching hogs: 1 cpu, 0 io, 0 vm, 0 hdd
stress: info: [59918] successful run completed in 20s
# Exit the job
(node)$> exit # OR CTRL-D
(access)$>
Exit htop in its terminal (press 'q' to exit) and press CTRL-D to disconnect to return to the access server.
Quit the interactive job by pressing CTRL-D to disconnect to return to the access server.
A First launcher (1 job, 1 task on one core)
We will create a first "launcher script", responsible to perform a single task over 1 core.
We will copy the default template provided by the ULHPC team for such usage, launcher.serial.sh.
(access)$> cd ~/tutorials/sequential # if not yet done
(access)$> cp ~/git/github.com/ULHPC/tutorials/sequential/basics/scripts/launcher.serial.sh . # <- trailing '.' means 'here
(access)$> mv launcher.serial.sh launcher.stressme-serial.sh
Use your favorite editor (nano, vim etc) to edit it as follows:
--- ~/git/github.com/ULHPC/tutorials/sequential/basics/scripts/launcher.stressme-serial.sh 2020-12-10 16:25:24.564580000 +0100
+++ launcher.stressme-serial.sh 2020-12-10 17:02:16.847752000 +0100
#! /bin/bash -l
############################################################################
# Default launcher for serial (one core) tasks
############################################################################
-###SBATCH -J Serial-jobname
+#SBATCH -J StressMe-single
@@ -32,7 +32,7 @@
# /!\ ADAPT TASK variable accordingly
# Absolute path to the (serial) task to be executed i.e. your favorite
# Java/C/C++/Ruby/Perl/Python/R/whatever program to be run
-TASK=${TASK:=${HOME}/bin/app.exe}
+TASK=${TASK:=${HOME}/tutorials/sequential/scripts/run_stressme}
Let's test it in an interactive job:
### Have an interactive job
# ... either directly
(access)$> si
# ... or using the HPC School reservation 'hpcschool'if needed - use 'sinfo -T' to check if active and its name
# (access)$> si --reservation=hpcschool
# check usage
(node)$> ./launcher.stressme-serial.sh -h
### DRY-RUN
(node)$> ./launcher.stressme-serial.sh -n
### Starting timestamp (s): 1607616148
/home/users/<login>/tutorials/sequential/scripts/run_stressme
### Ending timestamp (s): 1607616148"
# Elapsed time (s): 0
### REAL test
(node)$> ./launcher.stressme-serial.sh
### Starting timestamp (s): 1607616167
# /usr/bin/stress -c 1 -t 20
stress: info: [64361] dispatching hogs: 1 cpu, 0 io, 0 vm, 0 hdd
stress: info: [64361] successful run completed in 20s
### Ending timestamp (s): 1607616187"
# Elapsed time (s): 20
Hint: for the lazy persons, you can define on the fly the TASK variable as follows:
### DRY-RUN
(node)$> TASK=$(pwd)/scripts/run_stressme ./scripts/launcher.serial.sh -n
### Starting timestamp (s): 1607616148
/home/users/<login>/tutorials/sequential/scripts/run_stressme
### Ending timestamp (s): 1607616148"
# Elapsed time (s): 0
### REAL test
(node)$> TASK=$(pwd)/scripts/run_stressme ./scripts/launcher.serial.sh
### Starting timestamp (s): 1607616500
# /usr/bin/stress -c 1 -t 20
stress: info: [103707] dispatching hogs: 1 cpu, 0 io, 0 vm, 0 hdd
stress: info: [103707] successful run completed in 20s
### Ending timestamp (s): 1607616520"
# Elapsed time (s): 20
Quit your interactive job (exit or CTRL+D) and submit it as a passive job:
# Note: you may want/need to run it under the dedicated reservation set for the training event
# using sbatch --reservation=[...]
(access)$> sbatch ./launcher.stressme-serial.sh
Hint: still for the lazy persons, just run:
# Note: you may want/need to run it under the dedicated reservation set for the training event
# using [...] sbatch --reservation=[...]
# TASK=$(pwd)/scripts/run_stressme sbatch ./scripts/launcher.serial.sh
A VERY BAD StressMe Job Campaign: For loop on sbatch
Now the job campaign for N tasks (N=30) can be obtained by submitting N=30 jobs:
# Note: you may want/need to run it under the dedicated reservation set for the training event
# using sbatch --reservation=[...]
# /!\ ALWAYS echo your commands in a for loop before going real
(access)$> for i in $(seq 1 30); do echo sbatch ./launcher.stressme-serial.sh $i; done
sbatch ./launcher.stressme-serial.sh 1
sbatch ./launcher.stressme-serial.sh 2
sbatch ./launcher.stressme-serial.sh 3
[...]
sbatch ./launcher.stressme-serial.sh 28
sbatch ./launcher.stressme-serial.sh 29
sbatch ./launcher.stressme-serial.sh 30
# Equivalent of
# for i in {1..30}; do echo sbatch ./launcher.stressme-serial.sh $i; done
# Go real
(access)$> for i in $(seq 1 30); do sbatch ./launcher.stressme-serial.sh $i; done
Submitted batch job 2171802
Submitted batch job 2171803
Submitted batch job 2171804
Submitted batch job 2171805
Submitted batch job 2171806
Submitted batch job 2171807
Submitted batch job 2171809
Submitted batch job 2171810
Submitted batch job 2171811
Submitted batch job 2171812
Submitted batch job 2171813
Submitted batch job 2171814
Submitted batch job 2171815
Submitted batch job 2171816
Submitted batch job 2171817
Submitted batch job 2171818
Submitted batch job 2171819
Submitted batch job 2171820
Submitted batch job 2171821
Submitted batch job 2171822
Submitted batch job 2171823
Submitted batch job 2171824
Submitted batch job 2171825
Submitted batch job 2171827
Submitted batch job 2171828
Submitted batch job 2171829
Submitted batch job 2171830
Submitted batch job 2171831
Submitted batch job 2171832
Submitted batch job 2171834
Use sq to check the status of your job in the queue:
(access)$> sq # squeue -u $(whoami)
JOBID PARTIT QOS NAME USER NODE CPUS ST TIME TIME_LEFT PRIORITY NODELIST(REASON)
2171819 batch normal StressMe-single <login> 1 1 CG 0:19 59:41 12678 iris-081
2171820 batch normal StressMe-single <login> 1 1 CG 0:19 59:41 12678 iris-149
2171818 batch normal StressMe-single <login> 1 1 CG 0:18 59:42 12678 iris-081
2171831 batch normal StressMe-single <login> 1 1 R 0:13 59:47 12678 iris-075
2171832 batch normal StressMe-single <login> 1 1 R 0:13 59:47 12678 iris-075
2171834 batch normal StressMe-single <login> 1 1 R 0:13 59:47 12678 iris-075
2171830 batch normal StressMe-single <login> 1 1 R 0:15 59:45 12678 iris-075
2171825 batch normal StressMe-single <login> 1 1 R 0:16 59:44 12678 iris-082
2171827 batch normal StressMe-single <login> 1 1 R 0:16 59:44 12678 iris-082
2171828 batch normal StressMe-single <login> 1 1 R 0:16 59:44 12678 iris-082
2171829 batch normal StressMe-single <login> 1 1 R 0:16 59:44 12678 iris-082
2171823 batch normal StressMe-single <login> 1 1 R 0:18 59:42 12678 iris-149
2171824 batch normal StressMe-single <login> 1 1 R 0:18 59:42 12678 iris-149
2171821 batch normal StressMe-single <login> 1 1 R 0:19 59:41 12678 iris-149
2171822 batch normal StressMe-single <login> 1 1 R 0:19 59:41 12678 iris-149
Check the past jobs statistics upon completion:
(access)$> sacct -X --format User,JobID,partition%12,state,time,start,end,elapsed,nnodes,ncpus,nodelist
User JobID Partition State Timelimit Start End Elapsed NNodes NCPUS NodeList
------- ------------ ------------ ---------- ---------- ------------------- ------------------- ---------- -------- ---------- ---------------
<login> 2171802 batch COMPLETED 01:00:00 2020-12-10T17:42:38 2020-12-10T17:42:39 00:00:01 1 1 iris-075
<login> 2171803 batch COMPLETED 01:00:00 2020-12-10T17:42:41 2020-12-10T17:42:43 00:00:02 1 1 iris-075
<login> 2171804 batch COMPLETED 01:00:00 2020-12-10T17:42:41 2020-12-10T17:42:44 00:00:03 1 1 iris-075
<login> 2171805 batch COMPLETED 01:00:00 2020-12-10T17:42:41 2020-12-10T17:42:45 00:00:04 1 1 iris-075
<login> 2171806 batch COMPLETED 01:00:00 2020-12-10T17:42:41 2020-12-10T17:42:47 00:00:06 1 1 iris-075
<login> 2171807 batch COMPLETED 01:00:00 2020-12-10T17:42:41 2020-12-10T17:42:48 00:00:07 1 1 iris-075
<login> 2171809 batch COMPLETED 01:00:00 2020-12-10T17:42:41 2020-12-10T17:42:49 00:00:08 1 1 iris-075
<login> 2171810 batch COMPLETED 01:00:00 2020-12-10T17:42:41 2020-12-10T17:42:50 00:00:09 1 1 iris-075
<login> 2171811 batch COMPLETED 01:00:00 2020-12-10T17:42:44 2020-12-10T17:42:54 00:00:10 1 1 iris-076
<login> 2171812 batch COMPLETED 01:00:00 2020-12-10T17:42:44 2020-12-10T17:42:55 00:00:11 1 1 iris-076
<login> 2171813 batch COMPLETED 01:00:00 2020-12-10T17:42:44 2020-12-10T17:42:56 00:00:12 1 1 iris-076
<login> 2171814 batch COMPLETED 01:00:00 2020-12-10T17:42:44 2020-12-10T17:42:57 00:00:13 1 1 iris-076
<login> 2171815 batch COMPLETED 01:00:00 2020-12-10T17:42:44 2020-12-10T17:42:58 00:00:14 1 1 iris-076
<login> 2171816 batch COMPLETED 01:00:00 2020-12-10T17:42:45 2020-12-10T17:42:59 00:00:14 1 1 iris-081
<login> 2171817 batch COMPLETED 01:00:00 2020-12-10T17:42:46 2020-12-10T17:43:01 00:00:15 1 1 iris-081
<login> 2171818 batch COMPLETED 01:00:00 2020-12-10T17:42:46 2020-12-10T17:43:04 00:00:18 1 1 iris-081
<login> 2171819 batch COMPLETED 01:00:00 2020-12-10T17:42:47 2020-12-10T17:43:06 00:00:19 1 1 iris-081
<login> 2171820 batch COMPLETED 01:00:00 2020-12-10T17:42:47 2020-12-10T17:43:06 00:00:19 1 1 iris-149
<login> 2171821 batch COMPLETED 01:00:00 2020-12-10T17:42:48 2020-12-10T17:43:09 00:00:21 1 1 iris-149
<login> 2171822 batch COMPLETED 01:00:00 2020-12-10T17:42:48 2020-12-10T17:43:09 00:00:21 1 1 iris-149
<login> 2171823 batch COMPLETED 01:00:00 2020-12-10T17:42:49 2020-12-10T17:43:11 00:00:22 1 1 iris-149
<login> 2171824 batch COMPLETED 01:00:00 2020-12-10T17:42:49 2020-12-10T17:43:12 00:00:23 1 1 iris-149
<login> 2171825 batch COMPLETED 01:00:00 2020-12-10T17:42:51 2020-12-10T17:43:14 00:00:23 1 1 iris-082
<login> 2171827 batch COMPLETED 01:00:00 2020-12-10T17:42:51 2020-12-10T17:43:17 00:00:26 1 1 iris-082
<login> 2171828 batch COMPLETED 01:00:00 2020-12-10T17:42:51 2020-12-10T17:43:17 00:00:26 1 1 iris-082
<login> 2171829 batch COMPLETED 01:00:00 2020-12-10T17:42:51 2020-12-10T17:43:19 00:00:28 1 1 iris-082
<login> 2171830 batch COMPLETED 01:00:00 2020-12-10T17:42:52 2020-12-10T17:43:19 00:00:27 1 1 iris-075
<login> 2171831 batch COMPLETED 01:00:00 2020-12-10T17:42:54 2020-12-10T17:43:22 00:00:28 1 1 iris-075
<login> 2171832 batch COMPLETED 01:00:00 2020-12-10T17:42:54 2020-12-10T17:43:23 00:00:29 1 1 iris-075
<login> 2171834 batch COMPLETED 01:00:00 2020-12-10T17:42:54 2020-12-10T17:43:24 00:00:30 1 1 iris-075
As can be seen, the campaign was initiated at 17:42:38 to be completed for the latest job at 17:43:24: it took thus 46s to be completed. It's 90,1% improvement on the sequential completion time.
Note that you can quickly get the maximum value of the 7th column with sort -k 7 [-n] as follows:
(access)$> sacct -X --format User,JobID,partition%12,state,time,start,end,elapsed,nnodes,ncpus,nodelist --noheader | sort -k 7 | tail -n 2
Repeating the experience on 100 consecutive tests demonstrated a completion between 18:43:22 and 18:46:19, i.e. on less than 3 minutes (2m 57s or 177s). It corresponds to a 96,5% improvement on the sequential completion time. The below table summarizes the results:
| #tasks | #jobs | Seq. time | Optimal | for [...] sbatch [...] |
Seq. Time Decrease | #nodes |
|---|---|---|---|---|---|---|
| 1 | 1 | 1s | 1s | 1s | 0% | 1 (max 1) |
| 10 | 10 | 55s | 10s | 19s | 65% | 3 (max 10) |
| 30 | 30 | 465s | 30s | 46s | 90,1% | 5 (max 30) |
| 100 | 100s | 5050s | 100s | 177s | 96,5% | 10 (max 100) |
This works of course but this is generally against best-practices:
- to complete N (serial) tasks, you need to submit N jobs that could be spread on up to N different nodes.
- This induces an non-necessary overload of the scheduler for (generally) very short tasks
- Node coverage is sub-optimal
- your serial jobs can be spread on up to N different nodes
Imagine expanding the job campaign to 1000 or 10000 test cases if not more, you risk to degrade significantly the HPC environment (the scheduler will likely have trouble to manage so many short-live jobs).
To prevent this behaviour, We have thus limit the number of jobs allowed per user (see sqos).
The objective is to invite you gently to regroup your tasks per node in order to better exploit their many-core configuration (28 cores on iris, 128 on aion).
A BAD StressMe Job Campaign: job arrays
Slurm support Job Arrays, a mechanism for submitting and managing collections of similar jobs quickly and easily;
All jobs must have the same initial options (e.g. size, time limit, etc.) and you can limit the numner
You just need to specify the array index values using the --array or -a option of the sbatch command. You SHOULD set the maximum number of simultaneously running tasks from the job array using a "%" separator -- typically match the number of cores per node (28 on iris, 128 on aion). For example `--array=0-100%28" will limit the number of simultaneously running tasks from this job array to 28.
Job arrays will have several additional environment variable set:
$SLURM_ARRAY_JOB_IDwill be set to the first job ID of the array$SLURM_ARRAY_TASK_IDwill be set to the job array index value.$SLURM_ARRAY_TASK_COUNTwill be set to the number of tasks in the job array.
We will copy the previous launcher:
(access)$> cd ~/tutorials/sequential # if not yet done
(access)$> cp launcher.stressme-serial.sh launcher.stressme-jobarray.sh
Use your favorite editor (nano, vim etc) to edit it as follows:
@@ -14,7 +14,8 @@
#SBATCH --ntasks-per-node 1
#SBATCH -c 1 # multithreading per task : -c --cpus-per-task <n> request
#__________________________
-#SBATCH -o logs/%x-%j.out # log goes into logs/<jobname>-<jobid>.out
+#SBATCH --array 1-9%5
+#SBATCH -o logs/%x-%A_%a.out # log goes into logs/<jobname>-<masterID>_<taskID>.out
@@ -75,7 +74,7 @@
start=$(date +%s)
echo "### Starting timestamp (s): ${start}"
-${CMD_PREFIX} ${TASK} ${OPTS}
+${CMD_PREFIX} ${TASK} ${OPTS} ${SLURM_ARRAY_TASK_ID}
Now you can run the same job campaign as before with a single sbatch command:
(access)$> sbatch --array 1-30%28 ./launcher.stressme-jobarray.sh
The above command WON'T work: massive job arrays campaign were run in the past that used to overwhelm the slurm controllers. To avoid this behaviour to repeat, we drastically reduce the capabilities of job arrays:
(access)$> scontrol show config | grep -i maxarray
In short, Don't use job arrays!!!: you can do better with GNU Parallel
A better launcher (1 job, #cores tasks per node)
As advised, you are encouraged to aggregate several serial tasks within a single job to better exploit the many-core configuration of each nodes (28 or 128 cores per node) - this would clearly be beneficial compared to you laptop that probably have "only" 4 cores.
One natural way of doing so would be to aggregate these tasks within a single slurm launcher, start them in the background (i.e. as child processes) by using the ampersand & after a Bash command, and the wait command:
# Dangerous
TASK=run_stressme
for i in {1..30}; do
srun -n1 --exclusive -c 1 --cpu-bind=cores ${TASK} $i &
done
wait
The ampersand & spawns the command srun -n1 --exclusive -c 1 --cpu-bind=cores ${TASK} $i in the background and allows the loop to continue to the next iteration without waiting for this sub-process to finish.
This approach is dangerous and the location of the wait command can be optimized to match the number of tasks per nodes as ideally, you would target to execute your bag of tasks by groups of 28 (or 128) to match the hardware configuration of ULHPC nodes.
In generic terms, you wish to target ${SLURM_NTASKS_PER_NODE} (if set) or the output of nproc --all (28 on iris) process to be forked assuming you use a full node.
The code would be as follows:
# Better but still dangerous
TASK=run_stressme
ncores=${SLURM_NTASKS_PER_NODE:-$(nproc --all)}
For i in {1..30}; do
srun -n1 --exclusive -c 1 --cpu-bind=cores ${TASK} $i &
[[ $((i%ncores)) -eq 0 ]] && wait
done
wait
You can test it from the sample launcher launcher.serial-ampersand.sh.
(access)$> cd ~/tutorials/sequential # if not yet done
(access)$> cp ~/git/github.com/ULHPC/tutorials/sequential/basics/scripts/launcher.serial-ampersand.sh . # <- trailing '.' means 'here
(access)$> mv launcher.serial-ampersand.sh launcher.stressme-serial-ampersand.sh
Use your favorite editor (nano, vim etc) to edit it as follows:
--- scripts/launcher.serial-ampersand.sh 2020-12-12 17:05:02.346701013 +0100
+++ launcher.stressme-serial-ampersand.sh 2020-12-12 17:06:30.001896000 +0100
@@ -5,7 +5,7 @@
# within one node using the Bash & (ampersand), a builtin control operator
# used to fork processes, and the wait command.
############################################################################
-###SBATCH -J Serial-ampersand
+#SBATCH -J StressMe-ampersand
#SBATCH --time=0-01:00:00 # 1 hour
#SBATCH --partition=batch
#__________________________
@@ -24,7 +24,7 @@
# /!\ ADAPT TASK variable accordingly
# Absolute path to the (serial) task to be executed i.e. your favorite
# Java/C/C++/Ruby/Perl/Python/R/whatever program to be run
-TASK=${TASK:=${HOME}/bin/app.exe}
+TASK=${TASK:=${HOME}/tutorials/sequential/scripts/run_stressme}
MIN=1
MAX=30
NCORES=${SLURM_NTASKS_PER_NODE:-$(nproc --all)}
Now launch the job campaign as before with a single sbatch command:
# Note: you may want/need to run it under the dedicated reservation set for the training event
# using sbatch --reservation=[...]
(access)$> sbatch ./launcher.stressme-serial-ampersand.sh
Submitted batch job 2175666
# if you have not done it upon submission, you can correct it with (adapt accordingly):
# scontrol update jobid=<JOBID> reservationname=<name>
Hint: still for the lazy persons afraid of copies/modifications, just run:
# Note: you may want/need to run it under the dedicated reservation set for the training event
# using [...] sbatch --reservation=[...]
# TASK=$(pwd)/scripts/run_stressme sbatch ./scripts/launcher.serial-ampersand.sh
Check the duration of the job afterward:
# See all job steps with slist <jobID>
(access)$> slist 2175666
# sacct -j 2175666 --format User,JobID,Jobname%30,partition,state,time,elapsed,MaxRss,MaxVMSize,nnodes,ncpus,nodelist,AveCPU,ConsumedEnergyRaw
User JobID JobName Partition State Timelimit Elapsed MaxRSS MaxVMSize NNodes NCPUS NodeList AveCPU ConsumedEnergyRaw
--------- ------------ ------------------------------ ---------- ---------- ---------- ---------- ---------- ---------- -------- ---------- --------------- ---------- -----------------
svarrette 2175666 StressMe-ampersand batch COMPLETED 01:00:00 00:00:59 1 28 iris-121 11885
2175666.bat+ batch COMPLETED 00:00:59 7472K 178788K 1 28 iris-121 00:00:00 11850
2175666.ext+ extern COMPLETED 00:00:59 0 108056K 1 28 iris-121 00:00:00 11885
2175666.0 run_stressme COMPLETED 00:00:03 0 248544K 1 1 iris-121 00:00:00 949
2175666.1 run_stressme COMPLETED 00:00:04 0 248544K 1 1 iris-121 00:00:00 1279
2175666.2 run_stressme COMPLETED 00:00:01 0 248544K 1 1 iris-121 00:00:00 276
2175666.3 run_stressme COMPLETED 00:00:07 0 248544K 1 1 iris-121 00:00:00 1952
2175666.4 run_stressme COMPLETED 00:00:08 0 248544K 1 1 iris-121 00:00:00 2526
2175666.5 run_stressme COMPLETED 00:00:02 0 248544K 1 1 iris-121 00:00:00 635
[...]
2175666.22 run_stressme COMPLETED 00:00:20 0 248544K 1 1 iris-121 00:00:00 5395
2175666.23 run_stressme COMPLETED 00:00:25 0 248544K 1 1 iris-121 00:00:00 6434
2175666.24 run_stressme COMPLETED 00:00:18 0 248544K 1 1 iris-121 00:00:00 4930
2175666.25 run_stressme COMPLETED 00:00:23 0 248544K 1 1 iris-121 00:00:00 6027
2175666.26 run_stressme COMPLETED 00:00:24 0 248544K 1 1 iris-121 00:00:00 6228
2175666.27 run_stressme COMPLETED 00:00:28 0 248544K 1 1 iris-121 00:00:00 6944
2175666.28 run_stressme COMPLETED 00:00:30 0 248544K 1 1 iris-121 00:00:00 4413
2175666.29 run_stressme COMPLETED 00:00:30 392K 248544K 1 1 iris-121 00:00:29 4554
#
# seff 2175666
#
Job ID: 2175666
Cluster: iris
User/Group: svarrette/clusterusers
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 28
CPU Utilized: 00:07:45
CPU Efficiency: 28.15% of 00:27:32 core-walltime
Job Wall-clock time: 00:00:59
Memory Utilized: 7.30 MB
Memory Efficiency: 0.01% of 112.00 GB
You can get the aggregated statistics with -X:
# See aggregated statictis with slist <jobID> -X
(access)$> slist 2175666 -X
# sacct -j 2175666 --format User,JobID,Jobname%30,partition,state,time,elapsed,MaxRss,MaxVMSize,nnodes,ncpus,nodelist,AveCPU,ConsumedEnergyRaw -X
User JobID JobName Partition State Timelimit Elapsed MaxRSS MaxVMSize NNodes NCPUS NodeList AveCPU ConsumedEnergyRaw
--------- ------------ ------------------------------ ---------- ---------- ---------- ---------- ---------- ---------- -------- ---------- --------------- ---------- -----------------
svarrette 2175666 StressMe-ampersand batch COMPLETED 01:00:00 00:00:59 1 28 iris-121 11885
# [...]
The launcher script supports the option -c to limit the number of cores used.
Check it out it's effect on the htop terminal/session:
# Note: you may want/need to run it under the dedicated reservation set for the training event
# using sbatch --reservation=[...]
(access)$> sbatch ./launcher.stressme-serial-ampersand.sh -c 14
Finally, repeat with 100 and 10 tasks (aggreated within a single job):
# Note: you may want/need to run it under the dedicated reservation set for the training event
# using sbatch --reservation=[...]
(access)$> sbatch ./launcher.stressme-serial-ampersand.sh --max 100
Submitted batch job 2175664
(access)$> sbatch ./launcher.stressme-serial-ampersand.sh --max 10
Submitted batch job 2175665
# sq [...]
# Once completed, check all statistics - # /!\ ADAPT JobID accordingly
(access)$> slist 2175664,2175665,2175666 -X
# sacct -j 2175664,2175665,2175666 --format User,JobID,Jobname%30,partition,state,time,elapsed,MaxRss,MaxVMSize,nnodes,ncpus,nodelist,AveCPU,ConsumedEnergyRaw -X
User JobID JobName Partition State Timelimit Elapsed MaxRSS MaxVMSize NNodes NCPUS NodeList AveCPU ConsumedEnergyRaw
--------- ------------ ------------------------------ ---------- ---------- ---------- ---------- ---------- ---------- -------- ---------- --------------- ---------- -----------------
svarrette 2175664 StressMe-ampersand batch COMPLETED 01:00:00 00:04:33 1 28 iris-117 76260
svarrette 2175665 StressMe-ampersand batch COMPLETED 01:00:00 00:00:11 1 28 iris-121 1866
svarrette 2175666 StressMe-ampersand batch COMPLETED 01:00:00 00:00:59 1 28 iris-121 11885
Compared to the (VERY BAD) "for loop of sbatch" approach, we have thus drastically optimized the resources allocation (a single node allocated, when 10 and up to 100 were required for the bigger campaign (100 tasks)) at the price of a affordable time penalty, still largely beneficial when compared to sequential i.e.:
- for 30 tasks: 28% penalty (59s vs. 46s), still demonstrating a 87% improvement compared to the sequential run.
- for 100 tasks: 54% penalty (273s vs 177s), still demonstrating a 95% improvement compared to the sequential run.
Nevertheless, we had to revisit the logic of the launcher script and enforce several customization.
On the contrary, GNU Parallel allow flexibility and adaptability while minimizing the customization: it makes the pallalelization happen automagically based on the slurm constraits (--ntasks-per-node, -c).
We will focus on a single node run for reasons make clear later.
Best launcher based on GNU Parallel (1 job, 1 node, N tasks)
GNU Parallel is a tool for executing tasks in parallel, typically on a single machine. When coupled with the Slurm command srun, parallel becomes a powerful way of distributing a set of embarrassingly parallel tasks amongst a number of workers. This is particularly useful when the number of tasks is significantly larger than the number of available workers (i.e. $SLURM_NTASKS), and each tasks is independent of the others.
Follow now our GNU Parallel tutorial to become more accustomed with the special (complex) syntax of this tool.
To illustrate the advantages of this approach, we will use the generic GNU parallel launcher script designed by the UL HPC Team scripts/launcher.parallel.sh.
First copy this script template and make it your FINAL launcher for stressme:
(access)$> cd ~/tutorials/sequential # if not yet done
(access)$> cp ~/git/github.com/ULHPC/tutorials/sequential/basics/scripts/launcher.parallel.sh . # <- trailing '.' means 'here
(access)$> mv launcher.parallel.sh launcher.stressme.sh
Use your favorite editor (nano, vim etc) to edit it as follows:
--- scripts/launcher.parallel.sh 2020-12-12 14:28:56.978642778 +0100
+++ launcher.stressme.sh 2020-12-12 14:31:14.432175130 +0100
@@ -55,9 +55,9 @@
### /!\ ADAPT TASK and TASKLIST[FILE] variables accordingly
# Absolute path to the (serial) task to be executed i.e. your favorite
# Java/C/C++/Ruby/Perl/Python/R/whatever program to be run
-TASK=${TASK:=${HOME}/bin/app.exe}
+TASK=${TASK:=${HOME}/tutorials/sequential/scripts/run_stressme}
# Default task list, here {1..8} is a shell expansion interpreted as 1 2... 8
-TASKLIST="{1..8}"
+TASKLIST="{1..30}"
# TASKLISTFILE=path/to/input_file
# If non-empty, used as input source over TASKLIST.
TASKLISTFILE=
So not far from what you did form the basic first launcher.
Interactive test of ULHPC GNU Parallel launcher (exclusive job required)
You can test this launcher within an exclusive interactive job (otherwise the internal calls to srun --exclusive will conflict with the default settings of interactive jobs)
### Have an **exclusive** interactive job for 4 (embarrassingly parallel) tasks
# DON'T forget the exclusive flag for interactive tests of GNU parallel with our launcher
# ... either directly
(access)$> si --ntasks-per-node 4 --exclusive
# ... or using the HPC School reservation 'hpcschool'if needed - use 'sinfo -T' to check if active and its name
# (access)$> srun --reservation=hpcschool --ntasks-per-node 4 --exclusive --pty bash
As before, in another terminal (or another screen tab/windows), connect to that job and run htop.
Now you can make some tests:
# check usage
(node)$> ./launcher.stressme.sh -h # read [tf] manual, press Q to quit
################ DRY-RUN
(node)$> ./launcher.stressme.sh -n
### Starting timestamp (s): 1607707840
parallel --delay .2 -j 4 --joblog logs/state.parallel.log --resume srun --exclusive -n1 -c 1 --cpu-bind=cores /home/users/svarrette/tutorials/sequential/scripts/run_stressme {1} ::: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
### Ending timestamp (s): 1607707840"
# Elapsed time (s): 0
Beware that the GNU parallel option --resume makes it read the log file set by
--joblog (i.e. logs/state*.log) to figure out the last unfinished task (due to the
fact that the slurm job has been stopped due to failure or by hitting a walltime
limit) and continue from there.
In particular, if you need to rerun this GNU Parallel job, be sure to delete the
logfile logs/state*.parallel.log or it will think it has already finished!
############### TEST mode - parallel echo mode (always important to do before running effectively the commands)
(node)$> ./launcher.stressme.sh -t
### Starting timestamp (s): 1607708018
srun --exclusive -n1 -c 1 --cpu-bind=cores /home/users/svarrette/tutorials/sequential/scripts/run_stressme 1
srun --exclusive -n1 -c 1 --cpu-bind=cores /home/users/svarrette/tutorials/sequential/scripts/run_stressme 2
srun --exclusive -n1 -c 1 --cpu-bind=cores /home/users/svarrette/tutorials/sequential/scripts/run_stressme 3
[...]
srun --exclusive -n1 -c 1 --cpu-bind=cores /home/users/svarrette/tutorials/sequential/scripts/run_stressme 27
srun --exclusive -n1 -c 1 --cpu-bind=cores /home/users/svarrette/tutorials/sequential/scripts/run_stressme 28
srun --exclusive -n1 -c 1 --cpu-bind=cores /home/users/svarrette/tutorials/sequential/scripts/run_stressme 29
srun --exclusive -n1 -c 1 --cpu-bind=cores /home/users/svarrette/tutorials/sequential/scripts/run_stressme 30
### Ending timestamp (s): 1607708024"
# Elapsed time (s): 6
Beware that the GNU parallel option --resume makes it read the log file set by
--joblog (i.e. logs/state*.log) to figure out the last unfinished task (due to the
fact that the slurm job has been stopped due to failure or by hitting a walltime
limit) and continue from there.
In particular, if you need to rerun this GNU Parallel job, be sure to delete the
logfile logs/state*.parallel.log or it will think it has already finished!
/!\ WARNING: Test mode - removing sate file
############## REAL run
(node)$> ./launcher.stressme.sh
### Starting timestamp (s): 1607708111
# /usr/bin/stress -c 1 -t 1
stress: info: [127447] dispatching hogs: 1 cpu, 0 io, 0 vm, 0 hdd
stress: info: [127447] successful run completed in 1s
# /usr/bin/stress -c 1 -t 2
stress: info: [127439] dispatching hogs: 1 cpu, 0 io, 0 vm, 0 hdd
stress: info: [127439] successful run completed in 2s
[...]
# /usr/bin/stress -c 1 -t 29
stress: info: [128239] dispatching hogs: 1 cpu, 0 io, 0 vm, 0 hdd
stress: info: [128239] successful run completed in 29s
# /usr/bin/stress -c 1 -t 30
stress: info: [128276] dispatching hogs: 1 cpu, 0 io, 0 vm, 0 hdd
stress: info: [128276] successful run completed in 30s
### Ending timestamp (s): 1607708243"
# Elapsed time (s): 132
Beware that the GNU parallel option --resume makes it read the log file set by
--joblog (i.e. logs/state*.log) to figure out the last unfinished task (due to the
fact that the slurm job has been stopped due to failure or by hitting a walltime
limit) and continue from there.
In particular, if you need to rerun this GNU Parallel job, be sure to delete the
logfile logs/state*.parallel.log or it will think it has already finished!
A quick look in parallel on htop report in the second terminal/screen windows demonstrate the usage of only 4 cores as expressed in the slurm job (--ntasks-per-node 4):
IMPORTANT as highlighted by the ULHPC script: Beware that the GNU parallel option --resume makes it read the log file set by --joblog (i.e. logs/state*.log) to figure out the last unfinished task (due to the fact that the slurm job has been stopped due to failure or by hitting a walltime
limit) and continue from there.
In particular, as we now want to rerun the same GNU Parallel job, be sure to delete the
logfile logs/state*.parallel.log or your passive job will likely do nothing as it will think it has already finished!
(node)$> rm logs/state.parallel.log
You can even test on another set of parameters without changing your script:
# BEWARE of placing the range within surrounding double quotes!!!
(node)$> ./launcher.stressme.sh -n "{1..10}" # Dry-run
(node)$> ./launcher.stressme.sh -t "{1..10}" # Test
(node)$> ./launcher.stressme.sh "{1..10}" # Real run
(node)$> rm logs/state.parallel.log
To avoid cleaning the joblog file, you may want to exceptionally set it to /dev/null using the --joblog option also supported by our script.
Passive job submission
Now that you have validated the expected behavior of the launcher script (you may want to test on higher number of tasks per node: GNU parallel will just adapt without any change to the launcher script), it's time to go for a passive run at full capacity.
Exit htop in its terminal (press 'q' to exit) and press CTRL-D to disconnect to return to the access server.
Quit your interactive job (exit or CTRL+D) and submit it as a passive job:
# Exit the interactive job
(node)$> exit # OR CTRL-D
# Note: you may want/need to run it under the dedicated reservation set for the training event
# using sbatch --reservation=[...]
(access)$> sbatch ./launcher.stressme.sh # --joblog /dev/null
Hint: for the lazy persons, you can still define on the fly the TASK variable as follows:
(access)$> TASK=$(pwd)/scripts/run_stressme sbatch ./scripts/launcher.parallel.sh
A quick look in parallel on htop report (sq then sjoin <JOBID> then htop) in the second terminal/screen windows demonstrate the expected usage optimizing the full node.
As there are "only" 30 parameters tested by default and that the launcher is configured with 28 tasks per node, the time you open htop you will likely not see all cores used.
Yet you can repeat the experience for 100 run_stressme tasks quite conveniently, without changing the launcher script:
# Remove the state log
(access)$> rm logs/state.parallel.log
# Note: you may want/need to run it under the dedicated reservation set for the training event
# using sbatch --reservation=[...]
# BEWARE of placing the range within surrounding double quotes!!!
# --joblog option used **exceptionnally** to disable resume option
(access)$> sbatch ./launcher.stressme.sh --joblog /dev/null "{1..100}"
In this case, you can be reassured on your htop usage:
Note: if you use the default (old) version of parallel (GNU parallel 20160222), you may occasionally witness a single core that seems inactive. This is a bug inherent to that version (tied to the controlling process) corrected in more recent version.
Repeat with 10 tasks
# Note: you may want/need to run it under the dedicated reservation set for the training event
# using sbatch --reservation=[...]
# BEWARE of placing the range within surrounding double quotes!!!
(access)$> sbatch ./launcher.stressme.sh --joblog /dev/null "{1..10}"
And let's collect the aggregated statistics from these 3 jobs (adapt job ID accordingly):
(access)$> slist 2175717,2175719,2175721 -X
# sacct -j 2175717,2175719,2175721 --format User,JobID,Jobname%30,partition,state,time,elapsed,MaxRss,MaxVMSize,nnodes,ncpus,nodelist,AveCPU,ConsumedEnergyRaw -X
User JobID JobName Partition State Timelimit Elapsed MaxRSS MaxVMSize NNodes NCPUS NodeList AveCPU ConsumedEnergyRaw
--------- ------------ ------------------------------ ---------- ---------- ---------- ---------- ---------- ---------- -------- ---------- --------------- ---------- -----------------
svarrette 2175717 GnuParallel batch COMPLETED 01:00:00 00:00:13 1 28 iris-117 2430
svarrette 2175719 GnuParallel batch COMPLETED 01:00:00 00:00:36 1 28 iris-117 9311
svarrette 2175721 GnuParallel batch COMPLETED 01:00:00 00:03:57 1 28 iris-117 70702
[...]
Compared to the serial ampersand approach, we have thus obtained a significant improvement in time efficiency for the same node occupancy (1 single node allocated):
- for 30 tasks: 39% improvement (36s vs 59s), demonstrating a 92% improvement compared to the sequential run.
- it's also 22% better than the VERY BAD
for [...] sbatchapproach (36s vs 46s)
- it's also 22% better than the VERY BAD
- for 100 tasks: 13% improvement (237s vs. 277s), demonstrating a 95% improvement compared to the sequential run.
Embarrassingly [GNU] parallel tasks across multiples nodes
GNU Parallel supports the distribution of tasks to multiple compute nodes using ssh connections, i.e. via the the --sshloginfile <filename> or --sshlogin options.
However, Scaling Parallel with --sshlogin[file] is Not Recommended
Though this allows work to be balance between multiple nodes, past experience suggests that scaling is much less effective. And as the typical usage of GNU parallel includes embarrassingly parallel tasks, there is no real justification to go across multiples nodes with a single job.
For instance, let's assume you wish to explore the parameters {1..1000}, you can divide your search space in (for instance) 5 sub domains and dedicated one GnuParallel job (exploiting a full node) for each subdomain as follows:
(access)$> sbatch ./launcher.stressme.sh --joblog logs/state.1.parallel.log "{1..200}"
(access)$> sbatch ./launcher.stressme.sh --joblog logs/state.2.parallel.log "{201..400}"
(access)$> sbatch ./launcher.stressme.sh --joblog logs/state.3.parallel.log "{401..600}"
(access)$> sbatch ./launcher.stressme.sh --joblog logs/state.4.parallel.log "{601..800}"
(access)$> sbatch ./launcher.stressme.sh --joblog logs/state.5.parallel.log "{801..1000}"
Each of these jobs will cover (independently) the subdomain.
Note that you MUST set different joblog files to avoid collusion -- you can use the --joblog option supported by our launcher.
Finally, if you would need to scale to more than 5 such jobs, you are encouraged to use the job dependency mechanism implemented by Slurm to limit the number of concurrent running nodes.
This can be easily achieved with the singleton dependency (option -d of sbatch) and carefully selected job names (option -J of sbatch):
-d, --dependency=singleton: This job can begin execution after any previously launched jobs sharing the same job name and user have terminated. In other words, only one job by that name and owned by that user can be running or suspended at any point in time.
This would permit to restrict the number of concurrent jobs (leaving more resources for others)
For example, to explore {1..2000} (and more generally {$min..$max}), without exceeding concurrent running on more than 4 nodes (i.e. within a "tunnel" of 4 nodes max), you could proceed as follows:
# Abstract search space parameters
min=1
max=2000
chunksize=200
for i in $(seq $min $chunksize $max); do
${CMD_PREFIX} sbatch \
-J ${JOBNAME}_$(($i/$chunksize%${MAXNODES})) --dependency singleton \
${LAUNCHER} --joblog log/state.${i}.parallel.log "{$i..$((i+$chunksize))}";
done
A sample submission script scripts/submit_stressme_multinode is proposed to illustrate this concept:
(access)$> ./scripts/submit_stressme_multinode -h
Usage: submit_stressme_multinode [-x] [-N MAXNODES]
Sample submision script across multiple nodes
Execution won t spread on more than 4 nodes (singleton dependency)
-x --execute really submit the jobs with sbatch
-N --nodes MAXNODES set max. nodes
# Target restriction to 4 running nodes max
(access)$> ./scripts/submit_stressme_multinode
sbatch -J StressMe_0 --dependency singleton launcher.stressme.sh --joblog logs/state.1.parallel.log {1..201}
sbatch -J StressMe_1 --dependency singleton launcher.stressme.sh --joblog logs/state.201.parallel.log {201..401}
sbatch -J StressMe_2 --dependency singleton launcher.stressme.sh --joblog logs/state.401.parallel.log {401..601}
sbatch -J StressMe_3 --dependency singleton launcher.stressme.sh --joblog logs/state.601.parallel.log {601..801}
sbatch -J StressMe_0 --dependency singleton launcher.stressme.sh --joblog logs/state.801.parallel.log {801..1001}
# Target restriction to 2 running nodes max
(access)$> ./scripts/submit_stressme_multinode -N 2
sbatch -J StressMe_0 --dependency singleton launcher.stressme.sh --joblog logs/state.1.parallel.log {1..201}
sbatch -J StressMe_1 --dependency singleton launcher.stressme.sh --joblog logs/state.201.parallel.log {201..401}
sbatch -J StressMe_0 --dependency singleton launcher.stressme.sh --joblog logs/state.401.parallel.log {401..601}
sbatch -J StressMe_1 --dependency singleton launcher.stressme.sh --joblog logs/state.601.parallel.log {601..801}
sbatch -J StressMe_0 --dependency singleton launcher.stressme.sh --joblog logs/state.801.parallel.log {801..1001}
Now if you execute it, you will see only two running jobs and thus nodes (the other jobs are waiting for the dependency to be met)
(access)$> ./scripts/submit_stressme_multinode -N 2 -x
Submitted batch job 2175778
Submitted batch job 2175779
Submitted batch job 2175780
Submitted batch job 2175781
Submitted batch job 2175782
(access)$> sq
# squeue -u svarrette
JOBID PARTIT QOS NAME NODE CPUS ST TIME TIME_LEFT NODELIST(REASON)
2175780 batch normal StressMe_0 1 28 PD 0:00 1:00:00 (Dependency)
2175781 batch normal StressMe_1 1 28 PD 0:00 1:00:00 (Dependency)
2175782 batch normal StressMe_0 1 28 PD 0:00 1:00:00 (Dependency)
2175779 batch normal StressMe_1 1 28 R 0:02 59:58 iris-064
2175778 batch normal StressMe_0 1 28 R 0:05 59:55 iris-047