From 18ebfbfe76d22d17229b1e9474b6fad7fcf8a71c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bj=C3=B6rn=20Claremar?= <70746791+bclaremar@users.noreply.github.com> Date: Tue, 21 May 2024 15:28:29 +0200 Subject: [PATCH 01/12] Update devel.md --- docs/extra/devel.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/extra/devel.md b/docs/extra/devel.md index 8e0dbd84..3367b7f4 100644 --- a/docs/extra/devel.md +++ b/docs/extra/devel.md @@ -38,7 +38,7 @@ - A little cumbersome but doable! - For collaboration within a ``sens`` project your can have a "local" ``remote`` repo in your common project folder. -- [More on Git on Bianca](https://www.uppmax.uu.se/support/faq/software-faq/git-on-bianca/) +- [More on Git on Bianca](http://docs.uppmax.uu.se/cluster_guides/git_on_bianca/) From 0f6ab2c3240864377a679fc9bd304f754fcf249e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bj=C3=B6rn=20Claremar?= <70746791+bclaremar@users.noreply.github.com> Date: Tue, 21 May 2024 15:29:36 +0200 Subject: [PATCH 02/12] julia.md fix uppmax link --- docs/extra/julia.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/extra/julia.md b/docs/extra/julia.md index 280a1814..1f7c276e 100644 --- a/docs/extra/julia.md +++ b/docs/extra/julia.md @@ -24,7 +24,7 @@ - The first time Julia will precompile the package for you! - You may control the present "central library" by typing ``ml help julia/`` in the BASH shell. - There you will also find which python, gcc and openmpi version that are compatible. - - Or see the [Julia user guide at UPPMAX](https://www.uppmax.uu.se/support/user-guides/julia-user-guide/){:target="_blank"} + - Or see the [Julia user guide at UPPMAX](http://docs.uppmax.uu.se/software/julia/){:target="_blank"} - A possibly more up-to-date status can be found from the Julia shell: ``` julia From 23316981590a13127a27997dc3906af39443cf32 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bj=C3=B6rn=20Claremar?= <70746791+bclaremar@users.noreply.github.com> Date: Tue, 21 May 2024 15:33:34 +0200 Subject: [PATCH 03/12] slurm.md fix uppmax links --- docs/extra/slurm.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/extra/slurm.md b/docs/extra/slurm.md index 86642041..4a57d107 100644 --- a/docs/extra/slurm.md +++ b/docs/extra/slurm.md @@ -1,10 +1,10 @@ # More about Slurm -- [Link to Slurm session in Intro to UPPMAX course](https://www.uppmax.uu.se/digitalAssets/560/c_560271-l_1-k_uppmax-slurm-2023-02.pdf){:target="_blank"} +- [Link to Slurm session in Intro to UPPMAX course](https://www.uu.se/download/18.57591c9d18f3ec99a0521784/1715116006615/c_560271-l_1-k_uppmax-slurm-2024-01.pdf){:target="_blank"} - [Slurm documentation](https://slurm.schedmd.com/){:target="_blank"} -- [Slurm user guide](https://www.uppmax.uu.se/support/user-guides/slurm-user-guide/){:target="_blank"} -- [Discovering job resource usage with `jobstats`](https://www.uppmax.uu.se/support/user-guides/jobstats-user-guide/){:target="_blank"} -- [Plotting your core hour usage](https://www.uppmax.uu.se/support/user-guides/plotting-your-core-hour-usage/){:target="_blank"} +- [Slurm user guide](http://docs.uppmax.uu.se/cluster_guides/slurm/){:target="_blank"} +- [Discovering job resource usage with `jobstats`](http://docs.uppmax.uu.se/software/jobstats/){:target="_blank"} +- [Plotting your core hour usage](http://docs.uppmax.uu.se/software/projplot/){:target="_blank"} ## Example From 5cd77f3b849e2ee4fc61385e3e58fc06052e855c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bj=C3=B6rn=20Claremar?= <70746791+bclaremar@users.noreply.github.com> Date: Tue, 21 May 2024 15:36:48 +0200 Subject: [PATCH 04/12] slurm_intro_with_jobstats.md fix uppmax links --- docs/intermediate/slurm_intro_with_jobstats.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/intermediate/slurm_intro_with_jobstats.md b/docs/intermediate/slurm_intro_with_jobstats.md index cc93d743..b6a7b2f7 100644 --- a/docs/intermediate/slurm_intro_with_jobstats.md +++ b/docs/intermediate/slurm_intro_with_jobstats.md @@ -442,9 +442,9 @@ Examine the jobs run by user `douglas`. The relevant job numbers are the jobs wi ## Links - [Slurm documentation](https://slurm.schedmd.com/){:target="_blank"} -- [Slurm user guide](https://www.uppmax.uu.se/support/user-guides/slurm-user-guide/){:target="_blank"} -- [Discovering job resource usage with `jobstats`](https://www.uppmax.uu.se/support/user-guides/jobstats-user-guide/){:target="_blank"} -- [Plotting your core hour usage](https://www.uppmax.uu.se/support/user-guides/plotting-your-core-hour-usage/){:target="_blank"} +- [Slurm user guide](http://docs.uppmax.uu.se/cluster_guides/slurm/){:target="_blank"} +- [Discovering job resource usage with `jobstats`](http://docs.uppmax.uu.se/software/jobstats/){:target="_blank"} +- [Plotting your core hour usage](http://docs.uppmax.uu.se/software/projplot/){:target="_blank"} !!! abstract "Keypoints" From 552461c6dff43f9c244f3b102544887bf17d2514 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bj=C3=B6rn=20Claremar?= <70746791+bclaremar@users.noreply.github.com> Date: Tue, 21 May 2024 15:38:37 +0200 Subject: [PATCH 05/12] install.md fix links uppmax --- docs/intermediate/install.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/intermediate/install.md b/docs/intermediate/install.md index 69d8b69d..b58c3fde 100644 --- a/docs/intermediate/install.md +++ b/docs/intermediate/install.md @@ -47,9 +47,9 @@ - Usually it is not a problem to build on Rackham and move to Bianca. - ``cmake`` is available as module - Check with: ``$ ml avail cmake`` -- [Guide for compiling **serial** programs](https://www.uppmax.uu.se/support/user-guides/compiling-source-code/){:target="_blank"} -- [Guide for compiling **parallel** programs](https://www.uppmax.uu.se/support/user-guides/mpi-and-openmp-user-guide/){:target="_blank"} - - [Available **combinations** of compilers and parallel libraries](https://www.uppmax.uu.se/support/user-guides/mpi-and-openmp-user-guide/#tocjump_7075108295107558_2){:target="_blank"} +- [Guide for compiling **serial** programs](http://docs.uppmax.uu.se/cluster_guides/compiling_serial/){:target="_blank"} +- [Guide for compiling **parallel** programs](http://docs.uppmax.uu.se/cluster_guides/compiling_parallel/){:target="_blank"} + - [Available **combinations** of compilers and parallel libraries](http://docs.uppmax.uu.se/cluster_guides/compiling_parallel/#mpi-using-the-openmpi-library){:target="_blank"} ???- info "About CPU hardware on Bianca" From 8f2b9c4d708884ea661837ac0444c1c9953abd83 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bj=C3=B6rn=20Claremar?= <70746791+bclaremar@users.noreply.github.com> Date: Tue, 21 May 2024 15:41:53 +0200 Subject: [PATCH 06/12] Update .wordlist.txt --- .wordlist.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.wordlist.txt b/.wordlist.txt index d0c3b278..e82e527f 100644 --- a/.wordlist.txt +++ b/.wordlist.txt @@ -278,4 +278,4 @@ scontrol Snakemake TMP checkmark - +pseudonymisation From b102c69524ff5feb501d92e722e455e532e1214c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bj=C3=B6rn=20Claremar?= <70746791+bclaremar@users.noreply.github.com> Date: Tue, 21 May 2024 15:55:13 +0200 Subject: [PATCH 07/12] slurm_bianca.md table batch command --- docs/intermediate/slurm_bianca.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/intermediate/slurm_bianca.md b/docs/intermediate/slurm_bianca.md index ffc8ba81..51c8ae49 100644 --- a/docs/intermediate/slurm_bianca.md +++ b/docs/intermediate/slurm_bianca.md @@ -8,7 +8,23 @@ - Advanced job submission ## The Slurm Workload Manager +Free, popular, lightweight +Open source: +https://slurm.schedmd.com +available at al SNIC centra +UPPMAX Slurm userguide: +https://www.uppmax.uu.se/support/user-guides/slurm-user-guide/ + ### More on sbatch +Recap: + +sbatch | -A sens2023598 | -t 10:00 | -p core | -n 10 | my_job.sh +-|-|-|-|-|- +slurm batch| project name | max runtime | partition ("job type") | #cores | job script + + + + ### More on time limits ### Job walltime ### More on partitions From 0f2a1ac992d2872f4a0ded933f7016acce078509 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bj=C3=B6rn=20Claremar?= <70746791+bclaremar@users.noreply.github.com> Date: Tue, 21 May 2024 16:03:47 +0200 Subject: [PATCH 08/12] slurm_bianca.md partitions --- docs/intermediate/slurm_bianca.md | 36 +++++++++++++++++++++++++------ 1 file changed, 29 insertions(+), 7 deletions(-) diff --git a/docs/intermediate/slurm_bianca.md b/docs/intermediate/slurm_bianca.md index 51c8ae49..3f18bc9d 100644 --- a/docs/intermediate/slurm_bianca.md +++ b/docs/intermediate/slurm_bianca.md @@ -8,12 +8,11 @@ - Advanced job submission ## The Slurm Workload Manager -Free, popular, lightweight -Open source: -https://slurm.schedmd.com -available at al SNIC centra -UPPMAX Slurm userguide: -https://www.uppmax.uu.se/support/user-guides/slurm-user-guide/ + +- Free, popular, lightweight +- Open source: https://slurm.schedmd.com +- available at all SNIC centra +- [UPPMAX Slurm userguide](http://docs.uppmax.uu.se/cluster_guides/slurm/) ### More on sbatch Recap: @@ -23,11 +22,34 @@ sbatch | -A sens2023598 | -t 10:00 | -p core | -n 10 | my_job.sh slurm batch| project name | max runtime | partition ("job type") | #cores | job script +### More on time limits +- ``-t dd-hh:mm:ss`` +- ``0-00:10:00 = 00:10:00 = 10:00 = 10`` +- ``0-12:00:00 = 12:00:00`` +- ``3-00:00:00 = 3-0`` +- ``3-12:10:15`` -### More on time limits ### Job walltime + +???- question "When you have no idea how long a program will take to run, what should you book?" + + A: very long time, e.g. 10-00:00:00 + +???- question "When you have an idea of how long a program would take to run, what should you book?" + + A: overbook by 50% + ### More on partitions + +- ``-p core`` + - “core” is the default partition + - ≤ 16 cores on Bianca + - a script or program written without any thought on parallelism will use 1 core + +- ``-p node` + - if you wish to book full node(s) + ### Quick testing ### Debugging or complicated workflows ### Parameters in the job script or the command line? From 0a481cdca614f4686a32d07d4d97e28d47805360 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bj=C3=B6rn=20Claremar?= <70746791+bclaremar@users.noreply.github.com> Date: Tue, 21 May 2024 16:06:56 +0200 Subject: [PATCH 09/12] transfer.md fix https link to docs (sunet) --- docs/intermediate/transfer.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/intermediate/transfer.md b/docs/intermediate/transfer.md index 78f9fc7f..e9ee0a00 100644 --- a/docs/intermediate/transfer.md +++ b/docs/intermediate/transfer.md @@ -173,8 +173,7 @@ Exercise 1 and 2 are the most important, as: but do not wait for feedback. If you have no idea at all, read the linked UPPMAX documentation: - - What is [SUNET](docs.uppmax.uu.se/getting_started/get_inside_sunet)? - - What is [`ssh`](http://docs.uppmax.uu.se/software/ssh/)? What does it allow us to do? + - What is [SUNET](http://docs.uppmax.uu.se/getting_started/get_inside_sunet/)? - What is [`wharf`](http://docs.uppmax.uu.se/cluster_guides/wharf/)? What does it allow us to do? - What is [`rsync`](http://docs.uppmax.uu.se/software/rsync/)? - What is [`transit`](http://docs.uppmax.uu.se/cluster_guides/transit/)? From 1a9fd576cca43f7f7f545d270703cd4cde826c13 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Bj=C3=B6rn=20Claremar?= <70746791+bclaremar@users.noreply.github.com> Date: Tue, 21 May 2024 16:18:08 +0200 Subject: [PATCH 10/12] slurm_bianca.md more flags --- docs/intermediate/slurm_bianca.md | 69 +++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+) diff --git a/docs/intermediate/slurm_bianca.md b/docs/intermediate/slurm_bianca.md index 3f18bc9d..57ea5a6a 100644 --- a/docs/intermediate/slurm_bianca.md +++ b/docs/intermediate/slurm_bianca.md @@ -51,9 +51,56 @@ slurm batch| project name | max runtime | partition ("job type") | #cores | job - if you wish to book full node(s) ### Quick testing + +- The “devel” partition + + - max 2 nodes per job + - up to 1 hour in length + - only 1 at a time + - ``-p devcore``, ``-p devel` +???- question "Any free nodes in the devel partition? Check status with" + + - ``sinfo -p devel`` + - ``jobinfo -p devel` + +- more on these tools later +- High priority queue for short jobs + + - 4 nodes + - up to 15 minutes + - ``--qos=short`` + ### Debugging or complicated workflows +- Interactive jobs + + - handy for debugging a code or a script by executing it line by line or for using programs with a graphical user interface + - ``salloc -n 80 -t 03:00:00 -A sens2023598`` + - ``interactive -n 80 -t 03:00:00 -A sens2023598` + + - up to 12 hours + - useful together with the --begin=