Merge branch 'main' of https://github.com/UPPMAX/bianca_workshop

UPPMAX · May 24, 2024 · 3aec1a6 · 3aec1a6
2 parents ae9ac9a + 00d104d
commit 3aec1a6
Show file tree

Hide file tree

Showing 7 changed files with 51 additions and 184 deletions.
diff --git a/.wordlist.txt b/.wordlist.txt
@@ -284,3 +284,6 @@ centres
 OOM
 gres
 Formalisms
+hackathon
+snakemake
+nextflow
diff --git a/docs/extra/conda.md b/docs/extra/conda.md
@@ -1,13 +1,13 @@
 # Conda on Bianca
 
-!!! info "Objectives"
-    - This is a brief description of the necessary steps to use the local Conda repository at UPPMAX, and install things for yourself or your project using Conda. 
+???+ question "Read [Conda user guide](http://docs.uppmax.uu.se/cluster_guides/conda/)"
 
-!!! info "Hint"
-
-    - On Bianca, Conda is the first choice when installing packages, because there is a local mirror of most of the Conda repositories.
+    - Read [Conda user guide](http://docs.uppmax.uu.se/cluster_guides/conda/)
+        - skip [Working with Conda environments defined by files](http://docs.uppmax.uu.se/cluster_guides/conda/#working-with-conda-environments-defined-by-files)
+
+    - ONLY for the interested: [Working with Conda environments defined by files](http://docs.uppmax.uu.se/cluster_guides/conda/#working-with-conda-environments-defined-by-files)
+        - On bianca you have to get the `environments.yml` to wharf first!
 
-## Using Conda
 
 !!! info "Conda cheat sheet"
 
@@ -44,13 +44,13 @@
     -   Deactivate current environment: `conda deactivate`
 
 
-## Your conda settings on Bianca
 
-- ```export CONDA_ENVS_PATH=/a/path/to/a/place/in/your/project/```
+
+# Exercises
 
 !!! tip
 
-    - You may wnat to have the same path for all conda environments in the present project
+    - You may want to have the same path for all conda environments in the present project
     - ``echo "export CONDA_ENVS_PATH=/a/path/to/a/place/in/your/project/" >> ~/.bashrc`` 
         - Example: ``echo "export CONDA_ENVS_PATH=/proj/sens2023598/bjornc/conda" >> ~/.bashrc``
 
@@ -59,183 +59,32 @@
     - It seems you are required to use this path, ending with the name of your environment, together with ``--prefix`` when you install new envronments AND packages also after activating the conda environment!
     Like: ``conda install --prefix $CONDA_ENVS_PATH/<your-environment> ...``
 
-!!! tip
-
-    -   REMEMBER TO `conda clean -a` once in a while to remove unused and unnecessary files
-
-??? info "By choice"
-
-    - Run `source conda_init.sh` to initialise your shell (bash) to be able to run `conda activate` and `conda deactivate` etcetera instead of `source activate`. It will modify (append) your `.bashrc` file.
-
-    -   When conda is loaded you will by default be in the ``base`` environment, which works in the same way as other Conda environments. It is a “best practice” to avoid installing additional packages into your base software environment unless it is very general packages
-
-## Installing using Conda
-
-We have mirrored all major Conda repositories directly on UPPMAX, on
-both Rackham and Bianca. These are updated every third day. We have the
-following channels available:
-
-!!! info "Conda Channels"
-
-    -   bioconda
-    -   biocore
-    -   conda-forge
-    -   dranew
-    -   free
-    -   main
-    -   pro
-    -   qiime2
-    -   r
-    -   r2018.11
-    -   scilifelab-lts
-    -   nvidia
-    -   pytorch
-
-- You reach them all by loading the ``conda`` module.
-- You don't have to state the specific channel when using UPPMAX.
-- Also, you are offline on Bianca which means that the default is `--offline`, which you can specify if you want to simulate the experience on Rackham.
-
-!!! tip
-
-    - If you need a channel that isn't in our repository, we can easily add it. Just send us a message and we will do it.
 
+???+ question "Create a conda environment and install some packages"
 
-## Make a new conda environment
-
-!!! tip
-
-    - Since python or other packages are dependent on each-other expect solving the versions takes some time.
-    - use an interactive session!
-
-
-1.  Load Conda
-
-    ```bash 
-    module load conda
-    ```
-
-    - This grants you access to the latest version of Conda and all major repositories on all UPPMAX systems.
-    - Check the text output as ``conda`` is loaded, especially the first time, see below
-
+    -   First check the current installed packages while having `python/3.9.5` loaded
 
-3.  Create the Conda environment
+    -   Open a new terminal and have the old one available for later comparison
 
-    - Example:
+    -   Make sure ``python`` module is not active in the new terminal
 
-        ```bash
-        conda create --prefix  $CONDA_ENVS_PATH/python36-env python=3.6 numpy=1.13.1 matplotlib=2.2.2
-        ```
+    -   Start conda module
 
-    !!! info "The `mamba` alternative is not needed in newer versions of Conda"!
+    -  Make sure you have a folder in the project directory (`$USER` will automatically fill in you username. Handy!!)
 
-    - It all worked if you get something like this:
-
-        ```bash
-        # To activate this environment, use
-        #
-        #     $ conda activate python36-env 
-        #
-        # To deactivate an active environment, use
-        #
-        #     $ conda deactivate
-        ```
+    - ``mkdir /proj/sens2023598/$USER``
 
+    - ``mkdir /proj/sens2023598/$USER/conda``
 
-4.  Activate the conda environment by `source activate` if you have not enabled ``conda activate``, see above:
+    - Set a CONDA_ENVS_PATH
+        - Example: ``echo "export CONDA_ENVS_PATH=/proj/sens2023598/$USER/conda" >> ~/.bashrc``
 
-    ```bash 
-    source activate python36-env
-    ```
-
-    - You will see that your prompt is changing to start with `(python-36-env)` to show that you are within an environment.
 
-    - You can also see the installed packages by:
-
-    ```bash
-    conda list
-    pip list
-    ```
-    - you can also add more packages within the environment by exact version (use `=`) or latest (?) compatible version:
-    ```bash
-    conda install --prefix   $CONDA_ENVS_PATH/python36-env pandas
-    ```
-    - that may have given you ``pandas=1.1.5`` which would be the newest version compatible with ``python3.6`` and ``numpy=1.13.1``
+    -   Use the ``conda`` module on Bianca and the ``conda-forge`` channel to create an environment with name `bianca-course` with `python 3.7` and `numpy 1.15`
 
-5.  Now do your work!
-
-6.  Deactivate with ``conda deactivate`` (this will work in any case!)
-
-    ```bash
-    (python-36-env) $ conda deactivate
-    ```
-
-!!! warning
-
-    -   Conda is known to create **many** *small* files. Your diskspace is not only limited in GB:s, but also in number of files (typically `300000` in ``$home``).
-    -   Check your disk usage and quota limit with `uquota`
-    -   Do a `conda clean -a` once in a while to remove unused and unnecessary files
-
-
--   [More info about Conda on UPPMAX](http://docs.uppmax.uu.se/cluster_guides/conda/)
-
-## Working with Conda environments defined by files
-
--   Create an environment based on dependencies given in an environment
-    file:
-
-        $ conda env create --file environment.yml
-
--   Create file from present conda environment:
-
-        $ conda env export > environment.yml
-
-`environments.yml` (for conda) is a yaml-file which looks like this:
-
-```yaml
-name: my-environment
-channels:        # not needed on bianca
-- defaults
-dependencies:
-- numpy
-- matplotlib
-- pandas
-- scipy
-```
-
-`environments.yml` with versions:
-
-``` yaml
-name: my-environment
-channels:            #not needed on bianca
-- defaults
-dependencies:
-- python=3.7
-- numpy=1.18.1
-- matplotlib=3.1.3
-- pandas=1.1.2
-- scipy=1.6.2
-```
-
-!!! admonition "More on dependencies"
-
-    - Dependency management from course [Python for Scientific computing](https://aaltoscicomp.github.io/python-for-scicomp/dependencies/)
-
-# Exercises
-
-
-???+ question "Create a conda environment and install some packages"
-
-    -   First check the current installed packages while having `python/3.9.5` loaded
-
-    -   Open a new terminal and have the old one available for later comparison
-
-    -   Unload ``python`` module
-
-    -   Use the ``conda`` module on Rackham and create an environment with name `bianca-course` with `python 3.7` and `numpy 1.15`
-
-    -   Use your a path for `CONDA_ENVS_PATH` of your own choice (not doing this is perfectly OK and isnatalls in your ``$HOME`` folder) or `/proj/sens2023531/<user>/` 
+    -   Use your a path for `CONDA_ENVS_PATH`
         
-        -   (It may take a minute or so)
+        -   It may take a couple of minutes or so and do not forget to press `Y`when asked for!
 
     -   Activate!
 
@@ -253,26 +102,28 @@ dependencies:
     ``` sh
     $ module load conda
     ($ export CONDA_ENVS_PATH=/proj/sens2023598/$USER)
-    $ conda create --name HPC-python23 python=3.7 numpy=1.15
-    $ source activate HPC-python23
+    $ conda create -c conda-forge --prefix $CONDA_ENVS_PATH/bianca-course python=3.7 numpy=1.15
+    $ source activate bianca-course
     $ pip list
     $ python -V
     $ conda deactivate
     ```
 
+    - It should show numpy=1.15 among others and the python version 3.7.X for the conda env
+
 
 
 
 
 !!! abstract "keypoints"
 
-    -   Conda is an installer of packages but also bigger toolkits
+    - Conda is an installer of packages but also bigger toolkits
 
     - Conda on Bianca is easy since the repos in the most used channels are local.
 
-    -   Conda creates isolated environments not clashing with other installations of python and other versions of packages
+    -  Conda creates isolated environments not clashing with other installations of python and other versions of packages
 
     -   Conda environment requires that you install all packages needed by yourself, although automatically.
 
-    -   That is, you cannot load the python module and use the packages therein inside your Conda environment.
+    -   That is, you _cannot_ load the python module and use the packages therein inside your Conda environment.
 
diff --git a/docs/extra/pip.md b/docs/extra/pip.md
@@ -1,5 +1,9 @@
 # Install with pip to Bianca
 
+???+ question "Read through the content below"
+
+???+ question "Try to follow the steps with the package ``numpy==1.22.3`` for ``python/3.11.8`` "
+
 ## Check for packages 
 
 - from the Python shell with the ``import`` command

diff --git a/docs/extra/rpackages.md b/docs/extra/rpackages.md
@@ -1,5 +1,10 @@
 # Installing R packages on Bianca
 
+???+ question "Read through the content below"
+
+???+ question "Try to do the [exercise](rpackages.md#exercise-update-dowser)"
+
+
 ## First check if package is already in R_packages/x.y.z
 
 
@@ -66,19 +71,21 @@ drwxrwsr-x  7 douglas sw  4096 May 25 10:46 glmnetr
 
 
 
-## Example : Update dowser
+## Exercise: Update dowser
 
 [dowser on Readthedocs](https://dowser.readthedocs.io/en/latest/){:target="_blank"}
 
 !!! info
 
-    Dowser is part of the Immcantation analysis framework for Adaptive Immune Receptor Repertoire sequencing (AIRR-seq). Dowser provides a set of tools for performing phylogenetic analysis on B cell receptor repertoires. It supports building and visualizing trees using multiple methods, and implements statistical tests for discrete trait analysis of B cell migration, differentiation, and isotype switching.
+    - Dowser is part of the Immcantation analysis framework for Adaptive Immune Receptor Repertoire sequencing (AIRR-seq). 
+    - Dowser provides a set of tools for performing phylogenetic analysis on B cell receptor repertoires. 
+    - It supports building and visualizing trees using multiple methods, and implements statistical tests for discrete trait analysis of B cell migration, differentiation, and isotype switching.
 
 The version of dowser in ``R_packages/4.2.1`` is 1.1.0. It was updated to version 1.2.0 on [2023-05-30](https://cran.rstudio.com/web/packages/dowser/){:target="_blank"}.
 
 ### Install on Rackham
 
-You can update this for yourself by beginning on rackham. Do
+You can update this for yourself by beginning on **rackham**. Do
 
 ``` bash
 module load R_packages/4.2.1

diff --git a/docs/intermediate/install.md b/docs/intermediate/install.md
@@ -265,6 +265,8 @@ Then within R, try loading the package you want, like ``glmnet``:
     - [Extra material: Installing R packages](https://uppmax.github.io/bianca_workshop/extra/rpackages/){:target="_blank"}
     - [Extra material: Containers](https://uppmax.github.io/bianca_workshop/extra/containers/){:target="_blank"}
 
+    - One breakout room per topic: Help each-other!
+
 !!! discussion
 
     - Did it work out well?

diff --git a/docs/intermediate/slurm_bianca.md b/docs/intermediate/slurm_bianca.md
@@ -19,7 +19,7 @@
 
 !!! info "Lesson material in PDF"
 
-    - [PDF material](https://github.com/UPPMAX/bianca_workshop/blob/main/docs/presentations/Bianca-Slurm-2023-12-Diana.pdf)
+    - [PDF material](https://github.com/UPPMAX/bianca_workshop/blob/main/docs/presentations/Bianca-Slurm-2024-05-Diana.pdf)
 
 !!! info "See also"
 
@@ -225,7 +225,7 @@ sbatch -p devcore -t 00:15:00 jobscript.sh
     - check all your running jobs
     - what is the priority or your recently-submitted job?
     - submit a new job to run for 24h; note the job ID
-    - modify the name of the job to “wrongjob” and the maximum runtime to 7days, for example
+    - modify the name of the job to “wrongjob”
     - cancel your job with name “wrongjob”
 
 ## Determining job efficiency
@@ -401,7 +401,7 @@ env
 - Conceptually similar, but with different flavours
 - First define steps, each with an input, an output, and a command that transforms the input into output
 - Then just ask for the desired output and the system will handle the rest
-- Snakemake hackathon: https://www.naiss.se/event/online-training-snakemake-hackathon-2024-05/ (re-occuring event)
+- Snakemake hackathon: https://www.naiss.se/event/online-training-snakemake-hackathon-2024-05/ (re-occurring event)
 - Nextflow training: https://training.nextflow.io/
 
 ???+ question "Hands-on #4: make it your own"

diff --git a/docs/presentations/Bianca-Slurm-2024-05-Diana.pdf b/docs/presentations/Bianca-Slurm-2024-05-Diana.pdf