diff --git a/.gitignore b/.gitignore index f8a55749..2cf2d33a 100644 --- a/.gitignore +++ b/.gitignore @@ -1,4 +1,15 @@ +# Not sure what a .gitignore is? +# See: https://git-scm.com/docs/gitignore + +# These are directly copied from Jekyll's first-party docs on `.gitignore` files: +# https://jekyllrb.com/tutorials/using-jekyll-with-bundler/#commit-to-source-control + +# Ignore the default location of the built site, and caches and metadata generated by Jekyll *~ +_site/ +.sass-cache/ +.jekyll-cache/ +.jekyll-metadata __pycache__/ _build/ venv/ @@ -7,3 +18,8 @@ slides/theme/ slides/filter/ slides/convert.sh slides/*.html + + +# Ignore folders generated by Bundler +.bundle/ +vendor/ diff --git a/404.html b/404.html new file mode 100644 index 00000000..ec18bf77 --- /dev/null +++ b/404.html @@ -0,0 +1,11 @@ +--- +layout: default +title: 404 +permalink: /404 +nav_exclude: true +search_exclude: true +--- + +

Page not found

+ +

The page you requested could not be found. Please note that the structure of the CSC Computing Environment website was changed in February 2024. Try using the navigation {% if site.search_enabled != false %}or search {% endif %}to find what you're looking for or go to this site's home page.

diff --git a/Gemfile b/Gemfile index 37f5eaa4..04ab5ab3 100644 --- a/Gemfile +++ b/Gemfile @@ -1,2 +1,9 @@ source 'https://rubygems.org' + gem 'github-pages', group: :jekyll_plugins + +gem "jekyll", "~> 4.3.3" # installed by `gem jekyll` +# gem "webrick" # required when using Ruby >= 3 and Jekyll <= 4.2.2 + +gem "just-the-docs", "0.7.0" # pinned to the current release +# gem "just-the-docs" # always download the latest release diff --git a/LICENSE b/LICENSE index c1217d81..2f244ac8 100644 --- a/LICENSE +++ b/LICENSE @@ -1,4 +1,395 @@ -Copyright (C) 2020-2021 CSC - IT Center for Science Ltd. except where otherwise noted +Attribution 4.0 International -This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License, except where otherwise noted. -Full text of the license is available at . +======================================================================= + +Creative Commons Corporation ("Creative Commons") is not a law firm and +does not provide legal services or legal advice. Distribution of +Creative Commons public licenses does not create a lawyer-client or +other relationship. Creative Commons makes its licenses and related +information available on an "as-is" basis. Creative Commons gives no +warranties regarding its licenses, any material licensed under their +terms and conditions, or any related information. Creative Commons +disclaims all liability for damages resulting from their use to the +fullest extent possible. + +Using Creative Commons Public Licenses + +Creative Commons public licenses provide a standard set of terms and +conditions that creators and other rights holders may use to share +original works of authorship and other material subject to copyright +and certain other rights specified in the public license below. The +following considerations are for informational purposes only, are not +exhaustive, and do not form part of our licenses. + + Considerations for licensors: Our public licenses are + intended for use by those authorized to give the public + permission to use material in ways otherwise restricted by + copyright and certain other rights. Our licenses are + irrevocable. Licensors should read and understand the terms + and conditions of the license they choose before applying it. + Licensors should also secure all rights necessary before + applying our licenses so that the public can reuse the + material as expected. Licensors should clearly mark any + material not subject to the license. This includes other CC- + licensed material, or material used under an exception or + limitation to copyright. More considerations for licensors: + wiki.creativecommons.org/Considerations_for_licensors + + Considerations for the public: By using one of our public + licenses, a licensor grants the public permission to use the + licensed material under specified terms and conditions. If + the licensor's permission is not necessary for any reason--for + example, because of any applicable exception or limitation to + copyright--then that use is not regulated by the license. Our + licenses grant only permissions under copyright and certain + other rights that a licensor has authority to grant. Use of + the licensed material may still be restricted for other + reasons, including because others have copyright or other + rights in the material. A licensor may make special requests, + such as asking that all changes be marked or described. + Although not required by our licenses, you are encouraged to + respect those requests where reasonable. More_considerations + for the public: + wiki.creativecommons.org/Considerations_for_licensees + +======================================================================= + +Creative Commons Attribution 4.0 International Public License + +By exercising the Licensed Rights (defined below), You accept and agree +to be bound by the terms and conditions of this Creative Commons +Attribution 4.0 International Public License ("Public License"). To the +extent this Public License may be interpreted as a contract, You are +granted the Licensed Rights in consideration of Your acceptance of +these terms and conditions, and the Licensor grants You such rights in +consideration of benefits the Licensor receives from making the +Licensed Material available under these terms and conditions. + + +Section 1 -- Definitions. + + a. Adapted Material means material subject to Copyright and Similar + Rights that is derived from or based upon the Licensed Material + and in which the Licensed Material is translated, altered, + arranged, transformed, or otherwise modified in a manner requiring + permission under the Copyright and Similar Rights held by the + Licensor. For purposes of this Public License, where the Licensed + Material is a musical work, performance, or sound recording, + Adapted Material is always produced where the Licensed Material is + synched in timed relation with a moving image. + + b. Adapter's License means the license You apply to Your Copyright + and Similar Rights in Your contributions to Adapted Material in + accordance with the terms and conditions of this Public License. + + c. Copyright and Similar Rights means copyright and/or similar rights + closely related to copyright including, without limitation, + performance, broadcast, sound recording, and Sui Generis Database + Rights, without regard to how the rights are labeled or + categorized. For purposes of this Public License, the rights + specified in Section 2(b)(1)-(2) are not Copyright and Similar + Rights. + + d. Effective Technological Measures means those measures that, in the + absence of proper authority, may not be circumvented under laws + fulfilling obligations under Article 11 of the WIPO Copyright + Treaty adopted on December 20, 1996, and/or similar international + agreements. + + e. Exceptions and Limitations means fair use, fair dealing, and/or + any other exception or limitation to Copyright and Similar Rights + that applies to Your use of the Licensed Material. + + f. Licensed Material means the artistic or literary work, database, + or other material to which the Licensor applied this Public + License. + + g. Licensed Rights means the rights granted to You subject to the + terms and conditions of this Public License, which are limited to + all Copyright and Similar Rights that apply to Your use of the + Licensed Material and that the Licensor has authority to license. + + h. Licensor means the individual(s) or entity(ies) granting rights + under this Public License. + + i. Share means to provide material to the public by any means or + process that requires permission under the Licensed Rights, such + as reproduction, public display, public performance, distribution, + dissemination, communication, or importation, and to make material + available to the public including in ways that members of the + public may access the material from a place and at a time + individually chosen by them. + + j. Sui Generis Database Rights means rights other than copyright + resulting from Directive 96/9/EC of the European Parliament and of + the Council of 11 March 1996 on the legal protection of databases, + as amended and/or succeeded, as well as other essentially + equivalent rights anywhere in the world. + + k. You means the individual or entity exercising the Licensed Rights + under this Public License. Your has a corresponding meaning. + + +Section 2 -- Scope. + + a. License grant. + + 1. Subject to the terms and conditions of this Public License, + the Licensor hereby grants You a worldwide, royalty-free, + non-sublicensable, non-exclusive, irrevocable license to + exercise the Licensed Rights in the Licensed Material to: + + a. reproduce and Share the Licensed Material, in whole or + in part; and + + b. produce, reproduce, and Share Adapted Material. + + 2. Exceptions and Limitations. For the avoidance of doubt, where + Exceptions and Limitations apply to Your use, this Public + License does not apply, and You do not need to comply with + its terms and conditions. + + 3. Term. The term of this Public License is specified in Section + 6(a). + + 4. Media and formats; technical modifications allowed. The + Licensor authorizes You to exercise the Licensed Rights in + all media and formats whether now known or hereafter created, + and to make technical modifications necessary to do so. The + Licensor waives and/or agrees not to assert any right or + authority to forbid You from making technical modifications + necessary to exercise the Licensed Rights, including + technical modifications necessary to circumvent Effective + Technological Measures. For purposes of this Public License, + simply making modifications authorized by this Section 2(a) + (4) never produces Adapted Material. + + 5. Downstream recipients. + + a. Offer from the Licensor -- Licensed Material. Every + recipient of the Licensed Material automatically + receives an offer from the Licensor to exercise the + Licensed Rights under the terms and conditions of this + Public License. + + b. No downstream restrictions. You may not offer or impose + any additional or different terms or conditions on, or + apply any Effective Technological Measures to, the + Licensed Material if doing so restricts exercise of the + Licensed Rights by any recipient of the Licensed + Material. + + 6. No endorsement. Nothing in this Public License constitutes or + may be construed as permission to assert or imply that You + are, or that Your use of the Licensed Material is, connected + with, or sponsored, endorsed, or granted official status by, + the Licensor or others designated to receive attribution as + provided in Section 3(a)(1)(A)(i). + + b. Other rights. + + 1. Moral rights, such as the right of integrity, are not + licensed under this Public License, nor are publicity, + privacy, and/or other similar personality rights; however, to + the extent possible, the Licensor waives and/or agrees not to + assert any such rights held by the Licensor to the limited + extent necessary to allow You to exercise the Licensed + Rights, but not otherwise. + + 2. Patent and trademark rights are not licensed under this + Public License. + + 3. To the extent possible, the Licensor waives any right to + collect royalties from You for the exercise of the Licensed + Rights, whether directly or through a collecting society + under any voluntary or waivable statutory or compulsory + licensing scheme. In all other cases the Licensor expressly + reserves any right to collect such royalties. + + +Section 3 -- License Conditions. + +Your exercise of the Licensed Rights is expressly made subject to the +following conditions. + + a. Attribution. + + 1. If You Share the Licensed Material (including in modified + form), You must: + + a. retain the following if it is supplied by the Licensor + with the Licensed Material: + + i. identification of the creator(s) of the Licensed + Material and any others designated to receive + attribution, in any reasonable manner requested by + the Licensor (including by pseudonym if + designated); + + ii. a copyright notice; + + iii. a notice that refers to this Public License; + + iv. a notice that refers to the disclaimer of + warranties; + + v. a URI or hyperlink to the Licensed Material to the + extent reasonably practicable; + + b. indicate if You modified the Licensed Material and + retain an indication of any previous modifications; and + + c. indicate the Licensed Material is licensed under this + Public License, and include the text of, or the URI or + hyperlink to, this Public License. + + 2. You may satisfy the conditions in Section 3(a)(1) in any + reasonable manner based on the medium, means, and context in + which You Share the Licensed Material. For example, it may be + reasonable to satisfy the conditions by providing a URI or + hyperlink to a resource that includes the required + information. + + 3. If requested by the Licensor, You must remove any of the + information required by Section 3(a)(1)(A) to the extent + reasonably practicable. + + 4. If You Share Adapted Material You produce, the Adapter's + License You apply must not prevent recipients of the Adapted + Material from complying with this Public License. + + +Section 4 -- Sui Generis Database Rights. + +Where the Licensed Rights include Sui Generis Database Rights that +apply to Your use of the Licensed Material: + + a. for the avoidance of doubt, Section 2(a)(1) grants You the right + to extract, reuse, reproduce, and Share all or a substantial + portion of the contents of the database; + + b. if You include all or a substantial portion of the database + contents in a database in which You have Sui Generis Database + Rights, then the database in which You have Sui Generis Database + Rights (but not its individual contents) is Adapted Material; and + + c. You must comply with the conditions in Section 3(a) if You Share + all or a substantial portion of the contents of the database. + +For the avoidance of doubt, this Section 4 supplements and does not +replace Your obligations under this Public License where the Licensed +Rights include other Copyright and Similar Rights. + + +Section 5 -- Disclaimer of Warranties and Limitation of Liability. + + a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE + EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS + AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF + ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS, + IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION, + WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR + PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS, + ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT + KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT + ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU. + + b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE + TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION, + NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT, + INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES, + COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR + USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN + ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR + DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR + IN PART, THIS LIMITATION MAY NOT APPLY TO YOU. + + c. The disclaimer of warranties and limitation of liability provided + above shall be interpreted in a manner that, to the extent + possible, most closely approximates an absolute disclaimer and + waiver of all liability. + + +Section 6 -- Term and Termination. + + a. This Public License applies for the term of the Copyright and + Similar Rights licensed here. However, if You fail to comply with + this Public License, then Your rights under this Public License + terminate automatically. + + b. Where Your right to use the Licensed Material has terminated under + Section 6(a), it reinstates: + + 1. automatically as of the date the violation is cured, provided + it is cured within 30 days of Your discovery of the + violation; or + + 2. upon express reinstatement by the Licensor. + + For the avoidance of doubt, this Section 6(b) does not affect any + right the Licensor may have to seek remedies for Your violations + of this Public License. + + c. For the avoidance of doubt, the Licensor may also offer the + Licensed Material under separate terms or conditions or stop + distributing the Licensed Material at any time; however, doing so + will not terminate this Public License. + + d. Sections 1, 5, 6, 7, and 8 survive termination of this Public + License. + + +Section 7 -- Other Terms and Conditions. + + a. The Licensor shall not be bound by any additional or different + terms or conditions communicated by You unless expressly agreed. + + b. Any arrangements, understandings, or agreements regarding the + Licensed Material not stated herein are separate from and + independent of the terms and conditions of this Public License. + + +Section 8 -- Interpretation. + + a. For the avoidance of doubt, this Public License does not, and + shall not be interpreted to, reduce, limit, restrict, or impose + conditions on any use of the Licensed Material that could lawfully + be made without permission under this Public License. + + b. To the extent possible, if any provision of this Public License is + deemed unenforceable, it shall be automatically reformed to the + minimum extent necessary to make it enforceable. If the provision + cannot be reformed, it shall be severed from this Public License + without affecting the enforceability of the remaining terms and + conditions. + + c. No term or condition of this Public License will be waived and no + failure to comply consented to unless expressly agreed to by the + Licensor. + + d. Nothing in this Public License constitutes or may be interpreted + as a limitation upon, or waiver of, any privileges and immunities + that apply to the Licensor or You, including from the legal + processes of any jurisdiction or authority. + + +======================================================================= + +Creative Commons is not a party to its public +licenses. Notwithstanding, Creative Commons may elect to apply one of +its public licenses to material it publishes and in those instances +will be considered the β€œLicensor.” The text of the Creative Commons +public licenses is dedicated to the public domain under the CC0 Public +Domain Dedication. Except for the limited purpose of indicating that +material is shared under a Creative Commons public license or as +otherwise permitted by the Creative Commons policies published at +creativecommons.org/policies, Creative Commons does not authorize the +use of the trademark "Creative Commons" or any other trademark or logo +of Creative Commons without its prior written consent including, +without limitation, in connection with any unauthorized modifications +to any of its public licenses or any other arrangements, +understandings, or agreements concerning use of licensed material. For +the avoidance of doubt, this paragraph does not form part of the +public licenses. + +Creative Commons may be contacted at creativecommons.org. diff --git a/_config.yml b/_config.yml index f68cf921..00c38c84 100644 --- a/_config.yml +++ b/_config.yml @@ -1,15 +1,26 @@ title: CSC Computing Environment +description: Materials for the CSC Computing Environment -course +theme: just-the-docs + +url: https://csc-training.github.io/csc-env-eff +baseurl: "" plugins: - jekyll-sitemap -collections: - hands-on: - output: true +aux_links: + View on GitHub: https://github.com/csc-training/csc-env-eff + +favicon_ico: "/assets/images/logo.png" + +# Footer content +back_to_top: true +back_to_top_text: "Back to top" + +footer_content: "Copyright © 2020-2024 CSC – IT Center for Science Ltd. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 Unported License. This project has received funding from the European High-Performance Computing Joint Undertaking (EuroHPC JU) under grant agreements No 951732 and 101101903." -defaults: - - - scope: - path: "_hands-on" - values: - layout: "default" +gh_edit_link: true +gh_edit_link_text: "Edit this page on GitHub" +gh_edit_repository: "https://github.com/csc-training/csc-env-eff" +gh_edit_branch: "master" +gh_edit_view_mode: "edit" diff --git a/_hands-on/allas/README.md b/_hands-on/allas/README.md deleted file mode 100644 index 91e7acd6..00000000 --- a/_hands-on/allas/README.md +++ /dev/null @@ -1,8 +0,0 @@ -# Allas - -## Tutorials -* [Allas File transfer](tutorial_allas-file-transfer) -* [Allas File backup](tutorial_allas-file-backup) -* [Allas in Batch jobs](tutorial_allas-in-batch-jobs) -* [Allas mini tutorial](allas-mini-tutorial.md) This is an older version. -* [Allas tutorial](allas-tutorial.md) for deeper dive with bio-examples. \ No newline at end of file diff --git a/_hands-on/allas/allas-mini-tutorial.md b/_hands-on/allas/allas-mini-tutorial.md deleted file mode 100644 index 8c85ec9c..00000000 --- a/_hands-on/allas/allas-mini-tutorial.md +++ /dev/null @@ -1,189 +0,0 @@ ---- -topic: allas_old -title: Tutorial - A quicker look to Allas ---- - -# Allas Mini tutorial - -Open the view to the Allas service in your browser using the cPouta WWW-interface. - -Open: [https://pouta.csc.fi](https://pouta.csc.fi) - -And login with your csc user account. - -From the upper left corner, you find a project selection pop-up menu. If you have several projects available, select the -training project: **project_2004306** - -Then from the menu on left side of the interface, select: - -**Object Store -> Containers** - -And create new container by pressing button: **+Container** - -Keep the container _Not public_ and name it as 2004306_xxxx ( replace xxxx with your user account). - -Open the new bucket (that is here called as container) and upload one file from your computer. -Any file should do, but prefer a file that you can open in Puhti. - -During the exercises, you can use this interface to get another view to the buckets and objects in Allas. -> Note, that you need to **reload** the view in order to see the changes. - - -## Log in Puhti and use scratch - -1. Login to puhti.csc.fi and move to scratch: - -**Linux/Mac/MobaXterm on Windows** -```bash -ssh youcscusername@puhti.csc.fi (replace youcscusername with your csc user account) -``` - -**Windows/PuTTY** - - **host:** puhti.csc.fi - - **login as:** youcscusername (replace youcscusername with your csc user account) - - -In Puhti check your environment with command: -```bash -csc-workspaces -``` -Switch to the scratch directory of your project -```bash -cd /scratch/project_2004306# note! replace the text here (and below) with your project -``` -And create your own sub-directory, named after your training account (if this directory does not yet exist): -```bash -mkdir youcscusername -``` -(replace youcscusername with your user account) - -move to the directory. -```bash -cd youcscusername -``` - -## Using Allas - -Open connection to Allas. -```bash -module load allas -allas-conf -``` - -πŸ’‘ It might take a while with `module load allas`). - -If you have several Allas projects available, select the training project we are currently using. - -Study what you have in allas with commands -```bash -a-list -rclone lsd allas: - -a-list 2004306_xxxx -rclone ls allas:2004306_xxxx -``` - -Download the file you just uploaded to Allas from your local computer. -You can do that in two ways (replace _your-file-name_ with the name of the file you uploaded): -```bash -a-get 2004306_xxxx/your-file-name -``` -or -``` -rclone copy allas:2004306_xxxx/your-file-name ./ -``` - -Upload the file back to Allas. - -Try commands: - -```bash -a-put your-file-name -a-put --nc -b 2004306_xxxx -``` -Use use `a-put -h` to figure out the difference between the two commands above. - -Then do the upload with rclone: -```bash -rclone copy your-file-name allas:2004306_xxxx/ -``` -Locate the files you just uploaded in Pouta www-interface. - -## Using Allas in batch jobs - -The allas-conf command opens an Allas connection that is valid for eight hours. In the case of interactive usage this eight-hour limit is not problematic as allas-conf can be executed again to extend the validity of the connection. - -In the case of batch jobs, the situation is different, as it may take more than eight hours before the job even starts. -To be able to use Allas in a batch job, run _allas-conf_ again with option **-k**. -```bash -allas-conf -k -``` -Here the option -k indicates that the password will be stored in the environment variable $OS_PASSWORD. With this variable defined, you no longer need to define the password when you re-execute allas-conf with the -k option and the Allas project name. Note that if you mistype your password when using the -k option, you must use command **unset OS_PASSWORD** before you can try again. - -Refresh the connection with command: - -```bash -allas-conf -k project_2004306 -``` -When OS_PASSWORD is set, the a-commands (a-put, a-get, a-list, a-delete) automatically refresh the Allas connection when commands are ececuted in batch job. - -To test this, create a new batch job script. First open a new text file with command (relace xxxx here too): - -```bash -nano allas_xxxx.sh -``` -Then copy the batch job script sample lines from below to the text file you are editing. -Replace _xxxx_ with your user account and _your-file-name_ with the name of the file you -previously uploaded to Allas. When you are ready, save the changes with _Ctrl-o_ and exit nano with _Ctrl-x_ . - -```bash -#!/bin/bash -#SBATCH --job-name=my_allas_job -#SBATCH --account=project_2004306 -#SBATCH --time=00:05:00 -#SBATCH --mem-per-cpu=1G -#SBATCH --partition=test -#SBATCH --output=allas_output_%j.txt -#SBATCH --error=allas_errors_%j.txt - -a-get 2004306_xxxx/your-file-name -wc -l your-file-name > your-file-name.num_rows -a-put -b 2004306_xxxx --nc your-file-name.num_rows -``` -Submit the batch job with command: -```bash -sbatch allas_xxxx.sh -``` - -Use commands below to monitor the progress of your batch job. -```bash -squeue -u $USER -sacct -u $USER -a-list 2004306_xxxx -``` - -If you use rclone or swift instead of the a-commands, you need to add _source allas_conf_ commands to your script. -In this case, the batch job script for Puhti could look like: -```bash -#!/bin/bash -#SBATCH --job-name=my_allas_job -#SBATCH --account=project_2004306 -#SBATCH --time=00:05:00 -#SBATCH --mem-per-cpu=1G -#SBATCH --partition=test -#SBATCH --output=allas_output_%j.txt -#SBATCH --error=allas_errors_%j.txt - -#make sure connection to Allas is open -source /appl/opt/allas-cli-utils/allas_conf -f -k $OS_PROJECT_NAME -rclone copy allas:2004306_xxxx/your-file-name ./ - -wc -l your-file-name > your-file-name.num_rows - -#make sure connection to Allas is open -source /appl/opt/allas-cli-utils/allas_conf -f -k $OS_PROJECT_NAME -rclone copy your-file-name.num_rows allas:2004306_xxxx -``` - diff --git a/_hands-on/batch_jobs/README.md b/_hands-on/batch_jobs/README.md deleted file mode 100644 index 2896b1e2..00000000 --- a/_hands-on/batch_jobs/README.md +++ /dev/null @@ -1,13 +0,0 @@ -# Batch jobs - -## Tutorials -* [Serial batch job tutorial](serial.md) start with this. -* [Parallel batch job tutorial](parallel.md) using MPI and/or OpenMP. -* [Interactive batch job tutorial](interactive.md) on Puhti. -* [Hands-on batch jobs in Puhti tutorial](https://docs.csc.fi/support/tutorials/cmdline-handson/) A longer set of jobs to run. - -## Exercises -* [Retrieving bio data from repository](exercise_retrieving-bio-data.md) as an interactive job. (BIO) -* [Serial, array and parallel jobs with R + contours calculation from DEM with raster package (GIS) ](https://github.com/csc-training/geocomputing/tree/master/R/puhti) -* [Serial, array and parallel jobs with Python + NDVI calculation rasterio package (GIS) ](https://github.com/csc-training/geocomputing/tree/master/python/puhti) -* ... diff --git a/_hands-on/batch_resources/README.md b/_hands-on/batch_resources/README.md deleted file mode 100644 index 6ee73a68..00000000 --- a/_hands-on/batch_resources/README.md +++ /dev/null @@ -1,7 +0,0 @@ -# Batch job resource usage - -## Tutorials -* [Tutorial - sacct and seff](tutorial_sacct_and_seff.md) - -## Exercises -* [Exercise – sacct and seff](exercise_sacct_and_seff.md) diff --git a/_hands-on/connecting/README.md b/_hands-on/connecting/README.md deleted file mode 100644 index 30e64d7b..00000000 --- a/_hands-on/connecting/README.md +++ /dev/null @@ -1,11 +0,0 @@ -# Connecting to CSC Supercomputers - -## Tutorials -* [CSC account and project](credentials.md) -* [Login Puhti with ssh](ssh-puhti.md) -* [Create SSH-keys](ssh-keys.md) -* [Run R studio/Jupyter Notebook on Puhti via ssh-tunnel and browser](https://docs.csc.fi/support/tutorials/rstudio-or-jupyter-notebooks/) This requires ssh-keys (see above) and is documented here for legacy reasons. This way is superseded by the Open OnDemand based Puhti Web Interface, which provides the same functionality directly. (Advanced) - -## Exercises -* ... - diff --git a/_hands-on/disk-areas/README.md b/_hands-on/disk-areas/README.md deleted file mode 100644 index 6437d9c7..00000000 --- a/_hands-on/disk-areas/README.md +++ /dev/null @@ -1,9 +0,0 @@ -# Using disk areas for data storage in Puhti/Mahti - -## Tutorials -* [Using disk areas in Puhti](disk-areas-tutorial.md) - - -## Exercises -* [Main disk areas in Puhti](disk-areas-exercise-maindisks.md ) -* (Bonus)[Fast disk areas in Puhti](disk-areas-exercise-fastdisks.md ) diff --git a/_hands-on/disk-areas/disk-areas-exercise-maindisks.md b/_hands-on/disk-areas/disk-areas-exercise-maindisks.md deleted file mode 100644 index db8cfd85..00000000 --- a/_hands-on/disk-areas/disk-areas-exercise-maindisks.md +++ /dev/null @@ -1,66 +0,0 @@ ---- -topic: disk-areas-old -title: Advanced tutorial - Where to put files in CSC environment? ---- - -# Where to put files in CSC environment? - -## Binary and data files to share - -Where would you share your files such as programme packages and data files to other project members on the same supercomputer (i.e., on Puhti) as well as on Mahti (i.e, another supercomputer at CSC)? - -### Background - -This exercise is aimed at familiarising yourself with main disk areas in Puhti and Mahti supercomputers. Data files needed for computational analysis should be stored and shared in *scratch* directories and any software compilations and binaries should be shared in *proappl* directory. In order to find actual directories use commands such as `csc-workspaces` and `csc-projects`. Data transfer between two supercomputers can be done with many tools including `rsync`. In this example try to avoid using *allas* for data transfer between the supercomputers. You may download an example programme package (i.e.,ggplot2_3.3.3_Rprogramme.tar.gz) and data file (i.e., Merged.fasta) from [**allas** object storage](https://docs.csc.fi/data/Allas/) - -### Solution - -1. First login to Puhti supecomputer using *SSH*: - ```bash - ssh yourcscusername@puhti.csc.fi - ``` - -2. Download example programme package (i.e.,ggplot2_3.3.3_Rprogramme.tar.gz) and data file (i.e, Merged.fasta) from **allas** object storage - ```bash - wget https://a3s.fi/CSC_training/shared_files.tar.gz - tar -xavf shared_files.tar.gz - cd shared_files - ``` - - Let's assume that file *Merged.fasta* is data file intended for computational use and *ggplot2_3.3.3_Rprogramme.tar.gz* is a software tool needed for analysis. - You can now share file *Merged.fasta* in scratch folder and *ggplot2_3.3.3_Rprogramme.tar.gz* file in projapple directory. - -3. Share your *ggplot2_3.3.3_Rprogramme.tar.gz* file in *projappl* directory - - ```bash - cp ggplot2_3.3.3_Rprogramme.tar.gz /projappl/project_xxxx/$USER - - ``` -4. Share *Merged.fasta* file in *scratch* directory - - ```bash - cp Merged.fasta /scratch/project_xxxx/$USER - ``` - All new files and directories are also fully accessible for other group members (including read, write and execution permissions). If you want to restrict - access from your group members, you can reset the permissions with *chmod* command. - -5. Set read-only permissions for your group members for the file *Merged.fasta*: - - ```bash - cd /scratch/project_xxxx/$USER - chmod -R g-w Merged.fasta - ``` -5. sharing files in Mahti supercomputer - - you can copy *Merged.fasta* file on puhti to *scratch* drive on Mahti as below: - - ```bash - rsync -P Merged.fasta yourcscusername@mahti.csc.fi:/scratch/project_xxxx/$USER - ``` -6. you can copy *ggplot2_3.3.3_Rprogramme.tar.gz* file on puhti to *projappl* directory on Mahti as below: - - ```bash - rsync -P ggplot2_3.3.3_Rprogramme.tar.gz yourcscusername@mahti.csc.fi:/projappl/project_xxxx/$USER - ``` - - diff --git a/_hands-on/example_tutorial.md b/_hands-on/example_tutorial.md deleted file mode 100644 index 2508b960..00000000 --- a/_hands-on/example_tutorial.md +++ /dev/null @@ -1,101 +0,0 @@ ---- -topic: Put here the topic name that is found in ohter tutorials in the same folder -title: Title that is shown in the rendered index.md ---- - -# Descriptive Headline - -πŸ’¬ This document contains general styleguide for handson tutorials and exercises. - -> The prerequisites/learning outcomes: Sometimes there are many, but list if there's something essential but not obvious. - -1. Use numbered list to describe actual steps that a student does - ```bash - echo 'project_xxxx/yourcscusername' # Use commented code whenever clarification is necessary - ``` - - Try to get the code block indented nicely in hte rendered product -## Get started - -πŸ’¬ This is the first phase of this tutorial eg. a preparation. - -πŸ’‘ It is recommended to structurize the content with second level headlines. - -2. A code block can contain multiple lines - ```bash - export SCRATCH=/scratch/project_xxxx # Use xxxx for projectnumber - mkdir yourcscusername # Use yourcscusername if student should use their own - echo $USER # Or use the environment variable - ``` - -{:start="3"} -3. A step does not need to contain a code block, but instead make a point that student should notify - - Further explanation can be provided with intended list element - -☝🏻 A list element cannot start with indented block -- Meaning that it starts after regular text like this - -### 1. Further structurizing -1. Feel free to add more headlines if it supports the understandability of the content -2. The goal is to reduce the effort in following the material - -## Usage of Emojis - -πŸ’‘ The emojis are meant to structure text and to improve its readability. - -1. Look for the symbols-option for your computer - - Usually the emojis are there. -2. Copy the existing emojis if you don't find them anywhere else - -| Emoji | Purpose | -|-------|:--------| -| πŸ’¬ | General instructions or description | -| πŸ’­ | Provide extra information or insight | -| πŸ—― | Emphasise the message a bit | -| πŸ’‘ | Valuable information, ideas or suggestions | -| ☝🏻 | Notice or a side-note | -| ‼️ | Very important notice. Failing to comply with this will result in error of some kind | - -## Usage of images - -1. Add images to the `/slides/img/`-folder - - Or `/_hands-on/img/`-folder -2. Provide the path like this: -`![terminal-icon](../slides/img/terminal_icon1.png)` -3. The image should show like this: ![terminal-icon](../slides/img/terminal_icon1.png) - -## Providing job-files - -πŸ’¬ [Job-files](https://docs.csc.fi/computing/running/creating-job-scripts-puhti/#a-basic-batch-job-script) are scripts for SLURM-queueing system used in CSC supercomputers. - -1. Provide necessary job-files commented like this: - ```bash - #!/bin/bash - #SBATCH --job-name=test # Name of the job visible in the queue. - #SBATCH --account=project_xxxx # Choose the billing project. Has to be defined! - #SBATCH --partition=test # Job queues: test, interactive, small, large, longrun, hugemem, hugemem_longrun - #SBATCH --time=00:01:00 # Maximum duration of the job. Max: depends of the partition. - #SBATCH --mem=1G # How much RAM is reserved for job per node. - #SBATCH --ntasks=1 # Number of tasks. Max: depends on partition. - #SBATCH --cpus-per-task=1 # How many processors work on one task. Max: Number of CPUs per node. - - singularity exec tutorial.sif hello_world - ``` - -πŸ’­ For more information on batch jobs, please see [CSC Docs pages](https://docs.csc.fi/computing/running/getting-started/). - -## More information - -πŸ’‘ It is a good practice to provide more information on topic but it is not mandatory. - -πŸ’¬ This tutorial is meant as a brief introduction to get you started. - -πŸ’­ When searching the internet for instruction, pay attention that the instructions are for the same version of Singularity that you are using. There has been some command syntax changes etc. between the versions, so older instructions may not work with copy-paste. - -πŸ’‘ Always a good idea to link [CSC documentation](https://docs.csc.fi/). - -### About Markdown - -πŸ“ Ordered lists reset numbering if there's stuff in between. -Options: -- Manyally override numbering if needed with {:start="3"} -- Indent code blocks and they appear "under" the list item and the numbering goes on nicely \ No newline at end of file diff --git a/_hands-on/installing/README.md b/_hands-on/installing/README.md deleted file mode 100644 index c6cfc767..00000000 --- a/_hands-on/installing/README.md +++ /dev/null @@ -1,14 +0,0 @@ -# Installing software - -## Tutorials -* [Install MCL (a C code) from source in Puhti](installing_hands-on_mcl.md) -* [Compiling from source code - HPC use case](installing_high-performance-computing.md). This tutorial outlines the steps in installing C, C++, or Fortran code -* [Installing Java applictions](installing_hands-on_java.md) -* [Installing Perl applictions and libraries](installing_hands-on_perl.md) -* [Installing Python applications and libraries](installing_hands-on_python.md) -* [Installing R applictions and libraries](installing_hands-on_r.md) -* add more tutorials here... - -## Exercises -* TBA -* more exercises... \ No newline at end of file diff --git a/_hands-on/linux_prerequisites/README.md b/_hands-on/linux_prerequisites/README.md deleted file mode 100644 index 937b40ab..00000000 --- a/_hands-on/linux_prerequisites/README.md +++ /dev/null @@ -1,9 +0,0 @@ -# Linux prerequisites - -## Tutorials -* [Basic Linux commands](basic-linux-commands.md) -* [Basics of file editing](basic-file-editing.md) - - -## Exercises -* TBA diff --git a/_hands-on/linux_prerequisites/my-first-file.sh b/_hands-on/linux_prerequisites/my-first-file.sh deleted file mode 100644 index aa0a821f..00000000 --- a/_hands-on/linux_prerequisites/my-first-file.sh +++ /dev/null @@ -1,4 +0,0 @@ -# Hi there! Good job! -# This is a test text file. You can play around with this file. - -# You can try typing here with Nano, Emacs, Vim or another text editor! diff --git a/_hands-on/modules/README.md b/_hands-on/modules/README.md deleted file mode 100644 index 781c3256..00000000 --- a/_hands-on/modules/README.md +++ /dev/null @@ -1,8 +0,0 @@ -# Module system - -## Tutorials -* [Using modules in Puhti](modules-puhti.md) -* * [Biosoftwares in Puhti](module-exercise-with-aligners.md) - -## Exercises -* No exercises yet diff --git a/_hands-on/singularity/README.md b/_hands-on/singularity/README.md deleted file mode 100644 index bb5cc9ed..00000000 --- a/_hands-on/singularity/README.md +++ /dev/null @@ -1,14 +0,0 @@ -# Singularity - -## Tutorials -- [Running applications installed as containers](singularity-tutorial_running-installed.md) -- [Singularity tutorial](singularity-tutorial.md) - -## Extra tutorials -These tutorials require sudo accedd on the machine you -are running them, and can not be run on e.g. Puhti. -- [Creating Singularity containers](singularity_extra_creating-containers.md) -- [Replicating Conda environments as containers](singularity_extra_replicating-conda.md) - -## Exercises -- .. \ No newline at end of file diff --git a/_hands-on/throughput/README.md b/_hands-on/throughput/README.md deleted file mode 100644 index ce5d9bbe..00000000 --- a/_hands-on/throughput/README.md +++ /dev/null @@ -1,4 +0,0 @@ -# High throughput usage - -## Tutorials -* [Running Gaussian jobs using HyperQueue on Puhti](gaussian_hq.md) diff --git a/_includes/footer_custom.html b/_includes/footer_custom.html new file mode 100644 index 00000000..456cc832 --- /dev/null +++ b/_includes/footer_custom.html @@ -0,0 +1,9 @@ + +  + +  + + +{%- if site.footer_content -%} +

{{ site.footer_content }}

+{%- endif -%} \ No newline at end of file diff --git a/assets/images/eurocc.png b/assets/images/eurocc.png new file mode 100644 index 00000000..d7a58f60 Binary files /dev/null and b/assets/images/eurocc.png differ diff --git a/assets/images/logo.png b/assets/images/logo.png new file mode 100644 index 00000000..ebbd6f06 Binary files /dev/null and b/assets/images/logo.png differ diff --git a/index.md b/index.md index 1e99b9a1..c3020bbd 100644 --- a/index.md +++ b/index.md @@ -1,153 +1,27 @@ --- -title: CSC Computing Environment -author: CSC Training +title: Home +layout: home +nav_order: 1 --- -# Material for _CSC Computing Environment_ -course +# CSC Computing Environment +{: .fs-9 } -{% assign items = site.hands-on | sort: "title" | reverse %} +Welcome! This website contains materials for the *CSC Computing Environment* -course. +{: .fs-6 .fw-300 } -
- -
+## Guidelines -## 1. Prerequisites (accounts, connecting, command-line basics) -### 1.1 [Slides: Accounts and projects](https://a3s.fi/CSC_training/00_account_and_project.html) -### 1.2 [Video: Accounts and projects](https://video.csc.fi/media/t/0_j5ezfw80) -### 1.3 [Slides: Connecting to the CSC supercomputers](https://a3s.fi/CSC_training/01_logging_in.html) -### 1.4 [Video: Connecting to the CSC supercomputers](https://video.csc.fi/media/t/0_8fdqjuf9) -### 1.5 [Video tutorial: Connecting to Puhti](https://video.csc.fi/media/t/0_d0i4dk2j) -### 1.6 Tutorials and exercises -{% for hands-on in items %} -{% if hands-on.topic == 'connecting' %} -1. [{{ hands-on.title }}]({{ hands-on.url | relative_url }}) -{% endif %} -{% if hands-on.topic == 'Linux Prerequisites' %} -1. [{{ hands-on.title }}]({{ hands-on.url | relative_url }}) -{% endif %} -{% endfor %} +1. The course is divided into two parts and the material is organized by topics of increasing complexity -- Feel free to jump to Part 2 if you know the basics already! +1. In each topic, first read the slides / watch the video. +2. Complete the tutorial(s) to make sure you’ve got the steps right. +3. Try out the exercises to verify your new skills. +4. If you get stuck, consult [Docs CSC](https://docs.csc.fi) linked in the slides and the tutorials. +5. If the documentation does not provide a sufficient answer, please contact support by email or by filling in the contact form at . +6. Press and hold ctrl/cmd and click to open links in a new window or tab. +7. Left-click the slides to enable navigation with the arrow keys (or use the arrow icons in the bottom-right corner on each slide). +8. Click the home icon in the bottom-right corner on any slide to return to this page. +9. [A video with study tips](https://video.csc.fi/media/t/0_d7trmsru). -## 2. Introduction to the HPC environment -### 2.1 [Slides](https://a3s.fi/CSC_training/02_environment.html) -### 2.2 [Video: HPC environment](https://video.csc.fi/media/t/0_u1tvtzga) -### 2.3 [Video: CSC's datacenter in Kajaani](https://www.youtube.com/watch?v=HeqN0h391wg) - -## 3. Disk areas -### 3.1 [Slides](https://a3s.fi/CSC_training/03_disk_areas.html) -### 3.2 [Video: Disk areas](https://video.csc.fi/media/t/0_99zrt6or) -### 3.3 Tutorials and exercises -{% for hands-on in items %} -{% if hands-on.topic == 'disk-areas' %} -1. [{{ hands-on.title }}]({{ hands-on.url | relative_url }}) -{% endif %} -{% endfor %} - -## 4. Module system -### 4.1 [Slides](https://a3s.fi/CSC_training/04_modules.html) -### 4.2 [Video: Modules and preinstalled software](https://video.csc.fi/media/t/0_y57f260c) -### 4.5 Tutorials and exercises -{% for hands-on in items %} -{% if hands-on.topic == 'modules' %} -1. [{{ hands-on.title }}]({{ hands-on.url | relative_url }}) -{% endif %} -{% endfor %} - -## 5. Batch queue system and interactive use -### 5.1 [Slides](https://a3s.fi/CSC_training/05_batch_jobs.html) -### 5.2 [Video: Batch jobs](https://video.csc.fi/media/t/0_fagcet5x) -### 5.3 Tutorials and exercises -{% for hands-on in items %} -{% if hands-on.topic == 'Batch jobs' %} -1. [{{ hands-on.title }}]({{ hands-on.url | relative_url }}) -{% endif %} -{% endfor %} -1. [Exercise - Serial, array and parallel jobs with R + contours calculation from DEM with a raster package (GIS) ](https://github.com/csc-training/geocomputing/tree/master/R/puhti) -1. [Exercise - Serial, array and parallel jobs with Python + NDVI calculation rasterio package (GIS) ](https://github.com/csc-training/geocomputing/tree/master/python/puhti) - -## 6. Batch job resource usage -### 6.1 [Slides](https://a3s.fi/CSC_training/06_understanding_usage.html) -### 6.2 [Video: Resources usage](https://video.csc.fi/media/t/0_bfbgk9zf) -### 6.3 Tutorials and exercises -{% for hands-on in items %} -{% if hands-on.topic == 'batch resources' %} -1. [{{ hands-on.title }}]({{ hands-on.url | relative_url }}) -{% endif %} -{% endfor %} - -## 7. Allas and where to keep your data -### 7.1 [Slides](https://a3s.fi/CSC_training/07_allas.html) -### 7.2 [Video: Allas object storage](https://video.csc.fi/media/t/0_fvn4jbh1) -### 7.3 [Video: Using Allas](https://video.csc.fi/media/t/0_q04hc35j) -### 7.4 Tutorials and exercises -{% for hands-on in items %} -{% if hands-on.topic == 'allas' %} -1. [{{ hands-on.title }}]({{ hands-on.url | relative_url }}) -{% endif %} -{% endfor %} - -## 8. Installing your own software -### 8.1 [Slides](https://a3s.fi/CSC_training/08_installing.html) -### 8.2 [Video: Installing own software](https://video.csc.fi/media/t/0_anzwy1es) -### 8.3 Tutorials and exercises -{% for hands-on in items %} -{% if hands-on.topic == 'installing' %} -1. [{{ hands-on.title }}]({{ hands-on.url | relative_url }}) -{% endif %} -{% endfor %} - -## 9. Containers and Apptainer/Singularity -### 9.1 [Slides](https://a3s.fi/CSC_training/09_singularity.html) -### 9.2 [Video: Containers](https://video.csc.fi/media/t/0_0ws9ei53) -### 9.3 Tutorials and exercises -{% for hands-on in items %} -{% if hands-on.topic == 'singularity' %} -1. [{{ hands-on.title }}]({{ hands-on.url | relative_url }}) -{% endif %} -{% endfor %} - -## 10. How to speed up jobs -### 10.1 [Slides](https://a3s.fi/CSC_training/10_speed_up_jobs.html) -### 10.2 Tutorials and exercises -{% for hands-on in items %} -{% if hands-on.topic == 'throughput' %} -1. [{{ hands-on.title }}]({{ hands-on.url | relative_url }}) -{% endif %} -{% endfor %} - -## Information -

- -

-

- -

- This project has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 951732. - -

-
-
- -

- All materials (c) 2020-2023 by CSC – IT Center for Science Ltd.
- This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 Unported License, http://creativecommons.org/licenses/by-sa/4.0/ - -

-
-

-

 

+[Part 1 -- Get started with the basics!](part-1){: .btn .btn-primary .fs-5 .mb-4 .mb-md-0 .mr-2 } +[Part 2 -- Dig deeper into HPC workflows!](part-2){: .btn .btn-primary .fs-5 .mb-4 .mb-md-0 .mr-2 } diff --git a/js/slides-csc.js b/js/slides-csc.js deleted file mode 100644 index e450a8a2..00000000 --- a/js/slides-csc.js +++ /dev/null @@ -1,81 +0,0 @@ -const splitBeforeTags = ["H1", "H2", "H3"]; -let currentSlide = 0; - -function resizeRem() { - document.documentElement.style.setProperty( - '--screen-font-size', - document.body.childNodes[currentSlide].offsetWidth/45 + "px"); -} - -function hide(element) { - element.style.display = "none"; -} - -function show(element) { - element.style.display = "block"; -} - -function splitBefore(tag, tags) { - return tags.some(function(splitat){return splitat === tag;}); -} - -function chop(body) { - let islide = currentSlide; - body.insertBefore(document.createElement("div"), - body.childNodes[0]); - body.childNodes[0].className = "slide"; - hide(body.childNodes[0]); - while (body.childNodes.length > islide + 1) { - if (splitBefore(body.childNodes[islide + 1].tagName, splitBeforeTags) && body.childNodes[islide].childNodes.length > 1) { - body.insertBefore(body.childNodes[0].cloneNode(false), - body.childNodes[islide].nextSibling); - islide++; - } - body.childNodes[islide] - .appendChild(body.childNodes[islide + 1]); - } -} - -function init() { - var body = document.body; - chop(body); - show(body); - show(body.childNodes[currentSlide]); - resizeRem(); -} - -function previous() { - var body = document.body; - if (body.childNodes[currentSlide].previousSibling) { - hide(body.childNodes[currentSlide]); - currentSlide--; - show(body.childNodes[currentSlide]); - } -} - -function next() { - var body = document.body; - if (body.childNodes[currentSlide].nextSibling) { - hide(body.childNodes[currentSlide]); - currentSlide++; - show(body.childNodes[currentSlide]); - } -} - -function keypress(event) { - var key = event.charCode || event.keyCode; - if (key === 37 || key === 38) { - previous(); - } - if (key === 39 || key === 40) { - next(); - } -} - -function mouseclick(event) { - if ((event.clientX - document.body.clientWidth / 2) < 0) { - previous(); - } else { - next(); - } -} diff --git a/part-1/..md b/part-1/..md new file mode 100644 index 00000000..5cd5e679 --- /dev/null +++ b/part-1/..md @@ -0,0 +1,12 @@ +--- +layout: default +title: Basics +nav_order: 2 +has_children: true +permalink: /part-1 +--- + +# Part 1 -- Basics + +To make it as easy as possible to write documentation in plain Markdown, most UI components are styled using default Markdown elements with few additional CSS classes needed. +{: .fs-6 .fw-300 } diff --git a/_hands-on/allas/allas-tutorial.md b/part-1/allas/allas-tutorial.md similarity index 97% rename from _hands-on/allas/allas-tutorial.md rename to part-1/allas/allas-tutorial.md index e5195096..fee8353b 100644 --- a/_hands-on/allas/allas-tutorial.md +++ b/part-1/allas/allas-tutorial.md @@ -1,8 +1,15 @@ --- -topic: allas -title: Advanced Tutorial - Using Allas (bio-data example) +layout: default +title: Using Allas with bio data +parent: 7. Allas +grand_parent: Part 1 +nav_order: 4 +has_children: false +has_toc: false +permalink: /hands-on/allas/allas-tutorial.html --- + # Using Allas in CSC's HPC environment Before the actual exercise, open a view to the Allas service in your browser using the cPouta web interface. diff --git a/part-1/allas/index.md b/part-1/allas/index.md new file mode 100644 index 00000000..362a3194 --- /dev/null +++ b/part-1/allas/index.md @@ -0,0 +1,23 @@ +--- +layout: default +title: 7. Allas +parent: Part 1 +nav_order: 7 +has_children: true +has_toc: false +--- + +# 7. Allas and where to keep your data + +## [7.1 Slides](https://a3s.fi/CSC_training/07_allas.html) + +## [7.2 Video: Allas object storage](https://video.csc.fi/media/t/0_fvn4jbh1) + +## [7.3 Video: Using Allas](https://video.csc.fi/media/t/0_q04hc35j) + +## 7.4 Tutorials and exercises + +1. [Essential tutorial - Basic usage of Allas]({{ site.baseurl }}{% link part-1/allas/tutorial_allas-file-transfer.md %}) +2. [Tutorial - File backup with Allas]({{ site.baseurl }}{% link part-1/allas/tutorial_allas-file-backup.md %}) +3. [Tutorial - Allas in batch jobs]({{ site.baseurl }}{% link part-1/allas/tutorial_allas-in-batch-jobs.md %}) +4. [Advanced tutorial - Using Allas with bio data]({{ site.baseurl }}{% link part-1/allas/allas-tutorial.md %}) diff --git a/_hands-on/allas/tutorial_allas-file-backup.md b/part-1/allas/tutorial_allas-file-backup.md similarity index 90% rename from _hands-on/allas/tutorial_allas-file-backup.md rename to part-1/allas/tutorial_allas-file-backup.md index c51d4497..ce256d68 100644 --- a/_hands-on/allas/tutorial_allas-file-backup.md +++ b/part-1/allas/tutorial_allas-file-backup.md @@ -1,8 +1,15 @@ --- -topic: allas -title: Tutorial - File backup with Allas +layout: default +title: File backup with Allas +parent: 7. Allas +grand_parent: Part 1 +nav_order: 2 +has_children: false +has_toc: false +permalink: /hands-on/allas/tutorial_allas-file-backup.html --- + # Backing up files to Allas from Puhti ## Preparations (if not done already) diff --git a/_hands-on/allas/tutorial_allas-file-transfer.md b/part-1/allas/tutorial_allas-file-transfer.md similarity index 96% rename from _hands-on/allas/tutorial_allas-file-transfer.md rename to part-1/allas/tutorial_allas-file-transfer.md index 13df8ba4..78e3a3e4 100644 --- a/_hands-on/allas/tutorial_allas-file-transfer.md +++ b/part-1/allas/tutorial_allas-file-transfer.md @@ -1,6 +1,12 @@ --- -topic: allas -title: Tutorial - File transfer with Allas (essential) +layout: default +title: Basic usage of Allas +parent: 7. Allas +grand_parent: Part 1 +nav_order: 1 +has_children: false +has_toc: false +permalink: /hands-on/allas/tutorial_allas-file-transfer.html --- # Basic usage of Allas diff --git a/_hands-on/allas/tutorial_allas-in-batch-jobs.md b/part-1/allas/tutorial_allas-in-batch-jobs.md similarity index 96% rename from _hands-on/allas/tutorial_allas-in-batch-jobs.md rename to part-1/allas/tutorial_allas-in-batch-jobs.md index 1e9979c9..b72a1c23 100644 --- a/_hands-on/allas/tutorial_allas-in-batch-jobs.md +++ b/part-1/allas/tutorial_allas-in-batch-jobs.md @@ -1,8 +1,15 @@ --- -topic: allas -title: Tutorial - Allas in batch jobs +layout: default +title: Allas in batch jobs +parent: 7. Allas +grand_parent: Part 1 +nav_order: 3 +has_children: false +has_toc: false +permalink: /hands-on/allas/tutorial_allas-in-batch-jobs.html --- + # Using Allas in batch jobs ## Preparations diff --git a/_hands-on/batch_jobs/exercise_retrieving-bio-data.md b/part-1/batch-jobs/exercise_retrieving-bio-data.md similarity index 93% rename from _hands-on/batch_jobs/exercise_retrieving-bio-data.md rename to part-1/batch-jobs/exercise_retrieving-bio-data.md index ce8df6d3..72719511 100644 --- a/_hands-on/batch_jobs/exercise_retrieving-bio-data.md +++ b/part-1/batch-jobs/exercise_retrieving-bio-data.md @@ -1,8 +1,13 @@ --- -topic: Batch jobs -title: Exercise - Retrieving data from bio data repositories (advanced) +layout: default +title: Retrieving data from bio data repositories +parent: 5. Batch queue system and interactive use +grand_parent: Part 1 +nav_order: 4 +has_children: false +has_toc: false +permalink: /hands-on/batch_jobs/exercise_retrieving-bio-data.html --- - # Exercise: Retrieving data from bio data repositories > This exercise covers retrieving data from various commonly used bio data repositories. diff --git a/part-1/batch-jobs/index.md b/part-1/batch-jobs/index.md new file mode 100644 index 00000000..6a6ecb27 --- /dev/null +++ b/part-1/batch-jobs/index.md @@ -0,0 +1,23 @@ +--- +layout: default +title: 5. Batch queue system and interactive use +parent: Part 1 +nav_order: 5 +has_children: true +has_toc: false +--- + +# 5. Batch queue system and interactive use + +## [5.1 Slides](https://a3s.fi/CSC_training/05_batch_jobs.html) + +## [5.2 Video: Batch jobs](https://video.csc.fi/media/t/0_fagcet5x) + +## 5.3 Tutorials and exercises + +1. [Essential tutorial - Serial batch jobs]({{ site.baseurl }}{% link part-1/batch-jobs/serial.md %}) +2. [Tutorial - Parallel batch jobs]({{ site.baseurl }}{% link part-1/batch-jobs/parallel.md %}) +3. [Tutorial - Interactive batch jobs]({{ site.baseurl }}{% link part-1/batch-jobs/interactive.md %}) +4. [Exercise - Retrieving data from bio data repositories]({{ site.baseurl }}{% link part-1/batch-jobs/exercise_retrieving-bio-data.md %}) +5. [Exercise - Serial, array and parallel jobs with R + contours calculation from DEM with a raster package (GIS)](https://github.com/csc-training/geocomputing/tree/master/R/puhti) +6. [Exercise - Serial, array and parallel jobs with Python + NDVI calculation rasterio package (GIS)](https://github.com/csc-training/geocomputing/tree/master/python/puhti) diff --git a/_hands-on/batch_jobs/interactive.md b/part-1/batch-jobs/interactive.md similarity index 93% rename from _hands-on/batch_jobs/interactive.md rename to part-1/batch-jobs/interactive.md index 932003f8..ecc4e46d 100644 --- a/_hands-on/batch_jobs/interactive.md +++ b/part-1/batch-jobs/interactive.md @@ -1,6 +1,12 @@ --- -topic: Batch jobs -title: Tutorial - Interactive batch jobs +layout: default +title: Interactive batch jobs +parent: 5. Batch queue system and interactive use +grand_parent: Part 1 +nav_order: 3 +has_children: false +has_toc: false +permalink: /hands-on/batch_jobs/interactive.html --- # Batch job tutorial - Interactive jobs diff --git a/_hands-on/batch_jobs/parallel.md b/part-1/batch-jobs/parallel.md similarity index 97% rename from _hands-on/batch_jobs/parallel.md rename to part-1/batch-jobs/parallel.md index 7378b913..57ab6150 100644 --- a/_hands-on/batch_jobs/parallel.md +++ b/part-1/batch-jobs/parallel.md @@ -1,6 +1,12 @@ --- -topic: Batch jobs -title: Tutorial - Parallel batch jobs +layout: default +title: Parallel batch jobs +parent: 5. Batch queue system and interactive use +grand_parent: Part 1 +nav_order: 2 +has_children: false +has_toc: false +permalink: /hands-on/batch_jobs/parallel.html --- # Batch job tutorial - Parallel jobs diff --git a/_hands-on/batch_jobs/serial.md b/part-1/batch-jobs/serial.md similarity index 94% rename from _hands-on/batch_jobs/serial.md rename to part-1/batch-jobs/serial.md index 7b09d1b7..91d5b894 100644 --- a/_hands-on/batch_jobs/serial.md +++ b/part-1/batch-jobs/serial.md @@ -1,6 +1,12 @@ --- -topic: Batch jobs -title: Tutorial - Serial batch jobs (essential) +layout: default +title: Serial batch jobs +parent: 5. Batch queue system and interactive use +grand_parent: Part 1 +nav_order: 1 +has_children: false +has_toc: false +permalink: /hands-on/batch_jobs/serial.html --- # Batch job tutorial - Serial jobs diff --git a/_hands-on/batch_resources/exercise_sacct_and_seff.md b/part-1/batch-resources/exercise_sacct_and_seff.md similarity index 89% rename from _hands-on/batch_resources/exercise_sacct_and_seff.md rename to part-1/batch-resources/exercise_sacct_and_seff.md index 29f75eef..21a8eb37 100644 --- a/_hands-on/batch_resources/exercise_sacct_and_seff.md +++ b/part-1/batch-resources/exercise_sacct_and_seff.md @@ -1,7 +1,14 @@ --- -topic: batch resources -title: Exercise - Find your past job resource usage +layout: default +title: Find resource usage of recent jobs +parent: 6. Batch job resource usage +grand_parent: Part 1 +nav_order: 2 +has_children: false +has_toc: false +permalink: /hands-on/batch_resources/exercise_sacct_and_seff.html --- + # Get an overview of the resource usage of recent jobs ## What were my recent jobs? diff --git a/part-1/batch-resources/index.md b/part-1/batch-resources/index.md new file mode 100644 index 00000000..c7fa3958 --- /dev/null +++ b/part-1/batch-resources/index.md @@ -0,0 +1,19 @@ +--- +layout: default +title: 6. Batch job resource usage +parent: Part 1 +nav_order: 6 +has_children: true +has_toc: false +--- + +# 6. Batch job resource usage + +## [6.1 Slides](https://a3s.fi/CSC_training/06_understanding_usage.html) + +## [6.2 Video: Resource usage](https://video.csc.fi/media/t/0_bfbgk9zf) + +## 6.3 Tutorials and exercises + +1. [Essential tutorial - Understanding resource usage with `sacct` and `seff`]({{ site.baseurl }}{% link part-1/batch-resources/tutorial_sacct_and_seff.md %}) +2. [Exercise - Find your past job resource usage]({{ site.baseurl }}{% link part-1/batch-resources/exercise_sacct_and_seff.md %}) \ No newline at end of file diff --git a/_hands-on/batch_resources/tutorial_sacct_and_seff.md b/part-1/batch-resources/tutorial_sacct_and_seff.md similarity index 95% rename from _hands-on/batch_resources/tutorial_sacct_and_seff.md rename to part-1/batch-resources/tutorial_sacct_and_seff.md index 5fac5472..6d949b1a 100644 --- a/_hands-on/batch_resources/tutorial_sacct_and_seff.md +++ b/part-1/batch-resources/tutorial_sacct_and_seff.md @@ -1,6 +1,12 @@ --- -topic: batch resources -title: Tutorial - sacct and seff, resource usage (essential) +layout: default +title: Understanding resource usage with sacct and seff +parent: 6. Batch job resource usage +grand_parent: Part 1 +nav_order: 1 +has_children: false +has_toc: false +permalink: /hands-on/batch_resources/tutorial_sacct_and_seff.html --- # Using `sacct` and `seff` to understand resource usage of finished jobs diff --git a/_hands-on/disk-areas/disk-areas-exercise-fastdisks.md b/part-1/disk-areas/disk-areas-exercise-fastdisks.md similarity index 95% rename from _hands-on/disk-areas/disk-areas-exercise-fastdisks.md rename to part-1/disk-areas/disk-areas-exercise-fastdisks.md index bf60d1d9..3d29e5ea 100644 --- a/_hands-on/disk-areas/disk-areas-exercise-fastdisks.md +++ b/part-1/disk-areas/disk-areas-exercise-fastdisks.md @@ -1,6 +1,12 @@ --- -topic: disk-areas -title: Exercise - I/O intensive computing tasks (advanced) +layout: default +title: I/O intensive computing +parent: 3. Disk areas +grand_parent: Part 1 +nav_order: 4 +has_children: false +has_toc: false +permalink: /hands-on/disk-areas/disk-areas-exercise-fastdisks.html --- # How to run I/O intensive computing tasks efficiently? diff --git a/_hands-on/disk-areas/disk-areas-tutorial-fastdisks.md b/part-1/disk-areas/disk-areas-tutorial-fastdisks.md similarity index 95% rename from _hands-on/disk-areas/disk-areas-tutorial-fastdisks.md rename to part-1/disk-areas/disk-areas-tutorial-fastdisks.md index 3d07270f..702c1de5 100644 --- a/_hands-on/disk-areas/disk-areas-tutorial-fastdisks.md +++ b/part-1/disk-areas/disk-areas-tutorial-fastdisks.md @@ -1,6 +1,12 @@ --- -topic: disk-areas -title: Tutorial - Fast disk areas in CSC's computing environment +layout: default +title: Fast disk areas +parent: 3. Disk areas +grand_parent: Part 1 +nav_order: 3 +has_children: false +has_toc: false +permalink: /hands-on/disk-areas/disk-areas-tutorial-fastdisks.html --- # Fast disk areas in CSC's computing environment diff --git a/_hands-on/disk-areas/disk-areas-tutorial-lue.md b/part-1/disk-areas/disk-areas-tutorial-lue.md similarity index 94% rename from _hands-on/disk-areas/disk-areas-tutorial-lue.md rename to part-1/disk-areas/disk-areas-tutorial-lue.md index da6130a4..b1d4050c 100644 --- a/_hands-on/disk-areas/disk-areas-tutorial-lue.md +++ b/part-1/disk-areas/disk-areas-tutorial-lue.md @@ -1,8 +1,13 @@ --- -topic: disk-areas -title: Tutorial - Finding out where you have a lot of data (essential) +layout: default +title: Finding where you have lots of data +parent: 3. Disk areas +grand_parent: Part 1 +nav_order: 2 +has_children: false +has_toc: false +permalink: /hands-on/disk-areas/disk-areas-tutorial-lue.html --- - # Where do I have a lot of data? > In this tutorial you diff --git a/_hands-on/disk-areas/disk-areas-tutorial-maindisks.md b/part-1/disk-areas/disk-areas-tutorial-maindisks.md similarity index 96% rename from _hands-on/disk-areas/disk-areas-tutorial-maindisks.md rename to part-1/disk-areas/disk-areas-tutorial-maindisks.md index 48909977..5442a636 100644 --- a/_hands-on/disk-areas/disk-areas-tutorial-maindisks.md +++ b/part-1/disk-areas/disk-areas-tutorial-maindisks.md @@ -1,6 +1,12 @@ --- -topic: disk-areas -title: Tutorial - Main disk areas in CSC's computing environment (essential) +layout: default +title: Main disk areas +parent: 3. Disk areas +grand_parent: Part 1 +nav_order: 1 +has_children: false +has_toc: false +permalink: /hands-on/disk-areas/disk-areas-tutorial-maindisks.html --- # Where to store files in CSC's computing environment? diff --git a/part-1/disk-areas/index.md b/part-1/disk-areas/index.md new file mode 100644 index 00000000..b2b5f3d6 --- /dev/null +++ b/part-1/disk-areas/index.md @@ -0,0 +1,21 @@ +--- +layout: default +title: 3. Disk areas +parent: Part 1 +nav_order: 3 +has_children: true +has_toc: false +--- + +# 3. Disk areas + +## [3.1 Slides](https://a3s.fi/CSC_training/03_disk_areas.html) + +## [3.2 Video: Disk areas](https://video.csc.fi/media/t/0_99zrt6or) + +## 3.3 Tutorials and exercises + +1. [Essential tutorial - Main disk areas in CSC's computing environment]({{ site.baseurl }}{% link part-1/disk-areas/disk-areas-tutorial-maindisks.md %}) +2. [Essential tutorial - Finding out where you have a lot of data]({{ site.baseurl }}{% link part-1/disk-areas/disk-areas-tutorial-lue.md %}) +3. [Tutorial - Fast disk areas in CSC's computing environment]({{ site.baseurl }}{% link part-1/disk-areas/disk-areas-tutorial-fastdisks.md %}) +4. [Advanced exercise - I/O intensive computing tasks]({{ site.baseurl }}{% link part-1/disk-areas/disk-areas-exercise-fastdisks.md %}) diff --git a/part-1/hpc-intro/index.md b/part-1/hpc-intro/index.md new file mode 100644 index 00000000..50fc1505 --- /dev/null +++ b/part-1/hpc-intro/index.md @@ -0,0 +1,16 @@ +--- +layout: default +title: 2. Introduction to the HPC environment +parent: Part 1 +nav_order: 2 +has_children: false +has_toc: false +--- + +# 2. Introduction to the HPC environment + +## [2.1 Slides](https://a3s.fi/CSC_training/02_environment.html) + +## [2.2 Video: HPC environment](https://video.csc.fi/media/t/0_u1tvtzga) + +## [2.3 Video: CSC's national HPC infrastructure](https://www.youtube.com/watch?v=7OQSyNBh2yI) diff --git a/part-1/index.md b/part-1/index.md new file mode 100644 index 00000000..45346934 --- /dev/null +++ b/part-1/index.md @@ -0,0 +1,14 @@ +--- +layout: default +title: Part 1 +nav_order: 2 +has_children: true +permalink: /part-1 +--- + +# Part 1 -- Basics + +Start here to learn the basics of working efficiently in CSC's computing environment! This part +walks you through the prerequisites and introduces essential concepts such as disk areas, the +module system, batch jobs and the object storage service Allas. +{: .fs-6 .fw-300 } diff --git a/part-1/modules/index.md b/part-1/modules/index.md new file mode 100644 index 00000000..8d86d8d8 --- /dev/null +++ b/part-1/modules/index.md @@ -0,0 +1,20 @@ +--- +layout: default +title: 4. Module system +parent: Part 1 +permalink: /part-1/modules +nav_order: 4 +has_children: true +has_toc: false +--- + +# 4. Module system + +## [4.1 Slides](https://a3s.fi/CSC_training/04_modules.html) + +## [4.2 Video: Modules and preinstalled software](https://video.csc.fi/media/t/0_y57f260c) + +## 4.2 Tutorials and exercises + +1. [Essential tutorial - Modules in Puhti]({{ site.baseurl }}{% link part-1/modules/modules-puhti.md %}) +2. [Advanced tutorial - Biosoftware in Puhti]({{ site.baseurl }}{% link part-1/modules/module-exercise-with-aligners.md %}) diff --git a/_hands-on/modules/module-exercise-with-aligners.md b/part-1/modules/module-exercise-with-aligners.md similarity index 97% rename from _hands-on/modules/module-exercise-with-aligners.md rename to part-1/modules/module-exercise-with-aligners.md index 71aac4ea..3d49c682 100644 --- a/_hands-on/modules/module-exercise-with-aligners.md +++ b/part-1/modules/module-exercise-with-aligners.md @@ -1,6 +1,10 @@ --- -topic: modules -title: Advanced tutorial - Biosoftware in Puhti +layout: default +title: Biosoftware in Puhti +parent: 4. Module system +grand_parent: Part 1 +nav_order: 2 +permalink: /hands-on/modules/module-exercise-with-aligners.html --- # Biosoftware in Puhti diff --git a/_hands-on/modules/modules-puhti.md b/part-1/modules/modules-puhti.md similarity index 96% rename from _hands-on/modules/modules-puhti.md rename to part-1/modules/modules-puhti.md index 321f6876..0fb29cb8 100644 --- a/_hands-on/modules/modules-puhti.md +++ b/part-1/modules/modules-puhti.md @@ -1,6 +1,10 @@ --- -topic: modules -title: Tutorial - Modules in Puhti (essential) +layout: default +title: Modules in Puhti +parent: 4. Module system +grand_parent: Part 1 +nav_order: 1 +permalink: /hands-on/modules/modules-puhti.html --- # Modules in Puhti diff --git a/_hands-on/linux_prerequisites/basic-file-editing.md b/part-1/prerequisites/basic-file-editing.md similarity index 95% rename from _hands-on/linux_prerequisites/basic-file-editing.md rename to part-1/prerequisites/basic-file-editing.md index 4120987b..55eb2cc4 100644 --- a/_hands-on/linux_prerequisites/basic-file-editing.md +++ b/part-1/prerequisites/basic-file-editing.md @@ -1,6 +1,10 @@ --- -topic: Linux Prerequisites -title: Tutorial - Basic file editing +layout: default +title: Basic file editing +parent: 1. Prerequisites +grand_parent: Part 1 +nav_order: 4 +permalink: /hands-on/linux_prerequisites/basic-file-editing.html --- # Basic file editing diff --git a/_hands-on/linux_prerequisites/basic-linux-commands.md b/part-1/prerequisites/basic-linux-commands.md similarity index 94% rename from _hands-on/linux_prerequisites/basic-linux-commands.md rename to part-1/prerequisites/basic-linux-commands.md index b8e8ac2a..972e75a8 100644 --- a/_hands-on/linux_prerequisites/basic-linux-commands.md +++ b/part-1/prerequisites/basic-linux-commands.md @@ -1,6 +1,10 @@ --- -topic: Linux Prerequisites -title: Tutorial - Basic linux commands +layout: default +title: Basic Linux commands +parent: 1. Prerequisites +grand_parent: Part 1 +nav_order: 3 +permalink: /hands-on/linux_prerequisites/basic-linux-commands.html --- # Basic Linux commands diff --git a/_hands-on/connecting/credentials.md b/part-1/prerequisites/credentials.md similarity index 93% rename from _hands-on/connecting/credentials.md rename to part-1/prerequisites/credentials.md index 30ca31c9..80059956 100644 --- a/_hands-on/connecting/credentials.md +++ b/part-1/prerequisites/credentials.md @@ -1,6 +1,10 @@ --- -topic: connecting -title: Tutorial - Start with getting a CSC account and project (essential) +layout: default +title: CSC account and project +parent: 1. Prerequisites +grand_parent: Part 1 +nav_order: 1 +permalink: /hands-on/connecting/credentials.html --- # Get a CSC account and a project diff --git a/part-1/prerequisites/index.md b/part-1/prerequisites/index.md new file mode 100644 index 00000000..ee23eb6c --- /dev/null +++ b/part-1/prerequisites/index.md @@ -0,0 +1,29 @@ +--- +layout: default +title: 1. Prerequisites +parent: Part 1 +nav_order: 1 +permalink: /part-1/prerequisites +has_children: true +has_toc: false +--- + +# 1. Prerequisites (accounts, connecting, command-line basics) + +## [1.1 Slides: Accounts and projects](https://a3s.fi/CSC_training/00_account_and_project.html) + +## [1.2 Video: Accounts and projects](https://video.csc.fi/media/t/0_j5ezfw80) + +## [1.3 Slides: Connecting to CSC supercomputers](https://a3s.fi/CSC_training/01_logging_in.html) + +## [1.4 Video: Connecting to CSC supercomputer](https://video.csc.fi/media/t/0_8fdqjuf9) + +## [1.5 Video tutorial: Connecting to Puhti](https://video.csc.fi/media/t/0_d0i4dk2j) + +## 1.6 Tutorials and exercises + +1. [Essential tutorial - Getting a CSC account and project]({{ site.baseurl }}{% link part-1/prerequisites/credentials.md %}) +2. [Essential tutorial - Login to Puhti via a browser or SSH]({{ site.baseurl }}{% link part-1/prerequisites/ssh-puhti.md %}) +3. [Tutorial - Basic Linux commands]({{ site.baseurl }}{% link part-1/prerequisites/basic-linux-commands.md %}) +4. [Tutorial - Basic file editing]({{ site.baseurl }}{% link part-1/prerequisites/basic-file-editing.md %}) +5. [Advanced tutorial - Using SSH keys to authenticate connection]({{ site.baseurl }}{% link part-1/prerequisites/ssh-keys.md %}) \ No newline at end of file diff --git a/_hands-on/linux_prerequisites/my-first-file.txt b/part-1/prerequisites/my-first-file.txt similarity index 100% rename from _hands-on/linux_prerequisites/my-first-file.txt rename to part-1/prerequisites/my-first-file.txt diff --git a/_hands-on/connecting/ssh-keys.md b/part-1/prerequisites/ssh-keys.md similarity index 96% rename from _hands-on/connecting/ssh-keys.md rename to part-1/prerequisites/ssh-keys.md index 6a0a0081..5d9ddafd 100644 --- a/_hands-on/connecting/ssh-keys.md +++ b/part-1/prerequisites/ssh-keys.md @@ -1,6 +1,10 @@ --- -topic: connecting -title: Advanced tutorial - Using SSH keys to authenticate connection +layout: default +title: Using SSH keys +parent: 1. Prerequisites +grand_parent: Part 1 +nav_order: 5 +permalink: /hands-on/connecting/ssh-keys.html --- # Creating and adding SSH keys diff --git a/_hands-on/connecting/ssh-puhti.md b/part-1/prerequisites/ssh-puhti.md similarity index 98% rename from _hands-on/connecting/ssh-puhti.md rename to part-1/prerequisites/ssh-puhti.md index 860abe82..1a811ea8 100644 --- a/_hands-on/connecting/ssh-puhti.md +++ b/part-1/prerequisites/ssh-puhti.md @@ -1,6 +1,10 @@ --- -topic: connecting -title: Tutorial - Login to Puhti with a browser or SSH (essential) +layout: default +title: Login to Puhti +parent: 1. Prerequisites +grand_parent: Part 1 +nav_order: 2 +permalink: /hands-on/connecting/ssh-puhti.html --- # Log in to Puhti diff --git a/_hands-on/singularity/singularity-tutorial_part1.md b/part-2/containers/apptainer-tutorial-part1.md similarity index 95% rename from _hands-on/singularity/singularity-tutorial_part1.md rename to part-2/containers/apptainer-tutorial-part1.md index 2bd98538..7249f35c 100644 --- a/_hands-on/singularity/singularity-tutorial_part1.md +++ b/part-2/containers/apptainer-tutorial-part1.md @@ -1,6 +1,12 @@ --- -topic: singularity -title: Tutorial - Introduction to Apptainer (essential) +layout: default +title: Apptainer tutorial 1 +parent: 9. Containers and Apptainer +grand_parent: Part 2 +nav_order: 1 +has_children: false +has_toc: false +permalink: /hands-on/singularity/singularity-tutorial_part1.html --- # Apptainer tutorial diff --git a/_hands-on/singularity/singularity-tutorial_part2.md b/part-2/containers/apptainer-tutorial-part2.md similarity index 96% rename from _hands-on/singularity/singularity-tutorial_part2.md rename to part-2/containers/apptainer-tutorial-part2.md index 6088e4d3..52090d2c 100644 --- a/_hands-on/singularity/singularity-tutorial_part2.md +++ b/part-2/containers/apptainer-tutorial-part2.md @@ -1,6 +1,12 @@ --- -topic: singularity -title: Tutorial - Apptainer introduction continued +layout: default +title: Apptainer tutorial 2 +parent: 9. Containers and Apptainer +grand_parent: Part 2 +nav_order: 3 +has_children: false +has_toc: false +permalink: /hands-on/singularity/singularity-tutorial_part2.html --- # Apptainer tutorial, part 2 diff --git a/_hands-on/singularity/singularity_extra_creating-containers.md b/part-2/containers/creating-containers.md similarity index 96% rename from _hands-on/singularity/singularity_extra_creating-containers.md rename to part-2/containers/creating-containers.md index 6b1f0963..4fce2aa3 100644 --- a/_hands-on/singularity/singularity_extra_creating-containers.md +++ b/part-2/containers/creating-containers.md @@ -1,6 +1,12 @@ --- -topic: singularity -title: Exercise - Creating Apptainer containers +layout: default +title: Creating Apptainer containers +parent: 9. Containers and Apptainer +grand_parent: Part 2 +nav_order: 6 +has_children: false +has_toc: false +permalink: /hands-on/singularity/singularity_extra_creating-containers.html --- # Creating Apptainer containers diff --git a/_hands-on/singularity/singularity-tutorial_how_to_get_containers.md b/part-2/containers/getting-containers.md similarity index 93% rename from _hands-on/singularity/singularity-tutorial_how_to_get_containers.md rename to part-2/containers/getting-containers.md index abc368f0..9dc1fcf3 100644 --- a/_hands-on/singularity/singularity-tutorial_how_to_get_containers.md +++ b/part-2/containers/getting-containers.md @@ -1,6 +1,12 @@ --- -topic: singularity -title: Advanced tutorial - How to get containers +layout: default +title: How to get containers +parent: 9. Containers and Apptainer +grand_parent: Part 2 +nav_order: 4 +has_children: false +has_toc: false +permalink: /hands-on/singularity/singularity-tutorial_how_to_get_containers.html --- # How to get containers diff --git a/part-2/containers/index.md b/part-2/containers/index.md new file mode 100644 index 00000000..5118cfd7 --- /dev/null +++ b/part-2/containers/index.md @@ -0,0 +1,23 @@ +--- +layout: default +title: 9. Containers and Apptainer +parent: Part 2 +nav_order: 2 +has_children: true +has_toc: false +--- + +# 9. Containers and Apptainer + +## [9.1 Slides](https://a3s.fi/CSC_training/09_singularity.html) + +## [9.2 Video: Containers](https://video.csc.fi/media/t/0_0ws9ei53) + +## 9.3 Tutorials and exercises + +1. [Essential tutorial - Introduction to Apptainer]({{ site.baseurl }}{% link part-2/containers/apptainer-tutorial-part1.md %}) +2. [Tutorial - Apptainer introduction continued]({{ site.baseurl }}{% link part-2/containers/apptainer-tutorial-part2.md %}) +3. [Tutorial - Running containerized applications]({{ site.baseurl }}{% link part-2/containers/running-installed.md %}) +4. [Advanced tutorial - How to get containers]({{ site.baseurl }}{% link part-2/containers/getting-containers.md %}) +5. [Exercise - Replicating a Conda environment]({{ site.baseurl }}{% link part-2/containers/replicating-conda.md %}) +6. [Exercise - Creating Apptainer containers]({{ site.baseurl }}{% link part-2/containers/creating-containers.md %}) diff --git a/_hands-on/singularity/singularity_extra_replicating-conda.md b/part-2/containers/replicating-conda.md similarity index 95% rename from _hands-on/singularity/singularity_extra_replicating-conda.md rename to part-2/containers/replicating-conda.md index 4c2c8b5d..37d1779c 100644 --- a/_hands-on/singularity/singularity_extra_replicating-conda.md +++ b/part-2/containers/replicating-conda.md @@ -1,9 +1,15 @@ --- -topic: singularity -title: Exercise - Replicating a Conda environment +layout: default +title: Replicating a Conda environment +parent: 9. Containers and Apptainer +grand_parent: Part 2 +nav_order: 5 +has_children: false +has_toc: false +permalink: /hands-on/singularity/singularity_extra_replicating-conda.html --- -# Extra exercise: Replicating a Conda environment in a container +# Replicating a Conda environment in a container This is an extra exercise which can not be run on Puhti. You will need access to a computer or virtual machine where you have root privileges and that has Apptainer installed. diff --git a/_hands-on/singularity/singularity-tutorial_running-installed.md b/part-2/containers/running-installed.md similarity index 88% rename from _hands-on/singularity/singularity-tutorial_running-installed.md rename to part-2/containers/running-installed.md index 580f41ac..e24a9088 100644 --- a/_hands-on/singularity/singularity-tutorial_running-installed.md +++ b/part-2/containers/running-installed.md @@ -1,6 +1,12 @@ --- -topic: singularity -title: Tutorial - Running containerized applications +layout: default +title: Running containerized applications +parent: 9. Containers and Apptainer +grand_parent: Part 2 +nav_order: 3 +has_children: false +has_toc: false +permalink: /hands-on/singularity/singularity-tutorial_running-installed.html --- # Running containerized applications diff --git a/part-2/index.md b/part-2/index.md new file mode 100644 index 00000000..3a21326f --- /dev/null +++ b/part-2/index.md @@ -0,0 +1,14 @@ +--- +layout: default +title: Part 2 +nav_order: 3 +has_children: true +permalink: /part-2 +--- + +# Part 2 -- Next steps + +Ready to dive deeper? The second part covers more advanced topics, such as +working efficiently with large datasets (TBA), installing own software, containerization +and efficient HPC workflows. +{: .fs-6 .fw-300 } diff --git a/_hands-on/installing/installing_hands-on_binary.md b/part-2/installing/binary.md similarity index 94% rename from _hands-on/installing/installing_hands-on_binary.md rename to part-2/installing/binary.md index 748a66f1..50fbe6b5 100644 --- a/_hands-on/installing/installing_hands-on_binary.md +++ b/part-2/installing/binary.md @@ -1,6 +1,12 @@ --- -topic: installing -title: Tutorial - Installing binary applications (essential) +layout: default +title: Installing binary applications +parent: 8. Installing own software +grand_parent: Part 2 +nav_order: 1 +has_children: false +has_toc: false +permalink: /hands-on/installing/installing_hands-on_binary.html --- # Installing binary applications diff --git a/_hands-on/installing/installing_high-performance-computing.md b/part-2/installing/hpc.md similarity index 94% rename from _hands-on/installing/installing_high-performance-computing.md rename to part-2/installing/hpc.md index 9383fbef..83d2886b 100644 --- a/_hands-on/installing/installing_high-performance-computing.md +++ b/part-2/installing/hpc.md @@ -1,6 +1,12 @@ --- -topic: installing -title: Exercise - Installing own C, C++, or Fortran programs +layout: default +title: Installing own C, C++ or Fortran programs +parent: 8. Installing own software +grand_parent: Part 2 +nav_order: 7 +has_children: false +has_toc: false +permalink: /hands-on/installing/installing_hands-on_hpc.html --- # Installing and developing your own C, C++, or Fortran program in the CSC computing environment diff --git a/part-2/installing/index.md b/part-2/installing/index.md new file mode 100644 index 00000000..3f8d2e06 --- /dev/null +++ b/part-2/installing/index.md @@ -0,0 +1,24 @@ +--- +layout: default +title: 8. Installing own software +parent: Part 2 +nav_order: 1 +has_children: true +has_toc: false +--- + +# 8. Installing your own software + +## [8.1 Slides](https://a3s.fi/CSC_training/08_installing.html) + +## [8.2 Video: Installing own software](https://video.csc.fi/media/t/0_anzwy1es) + +## 8.3 Tutorials and exercises + +1. [Essential tutorial - Installing binary applications]({{ site.baseurl }}{% link part-2/installing/binary.md %}) +2. [Essential tutorial - Installing Python packages and environments]({{ site.baseurl }}{% link part-2/installing/python.md %}) +3. [Tutorial - Installing R applications and libraries]({{ site.baseurl }}{% link part-2/installing/r.md %}) +4. [Tutorial - Installing a simple C code from source]({{ site.baseurl }}{% link part-2/installing/mcl.md %}) +5. [Tutorial - Installing Perl applications and libraries]({{ site.baseurl }}{% link part-2/installing/perl.md %}) +6. [Tutorial - Installing Java applications]({{ site.baseurl }}{% link part-2/installing/java.md %}) +7. [Exercise - Installing own C, C++ or Fortran programs]({{ site.baseurl }}{% link part-2/installing/hpc.md %}) diff --git a/_hands-on/installing/installing_hands-on_java.md b/part-2/installing/java.md similarity index 92% rename from _hands-on/installing/installing_hands-on_java.md rename to part-2/installing/java.md index f89fc267..2d2cacbb 100644 --- a/_hands-on/installing/installing_hands-on_java.md +++ b/part-2/installing/java.md @@ -1,6 +1,12 @@ --- -topic: installing -title: Tutorial - Installing Java applications +layout: default +title: Installing Java applications +parent: 8. Installing own software +grand_parent: Part 2 +nav_order: 6 +has_children: false +has_toc: false +permalink: /hands-on/installing/installing_hands-on_java.html --- # Installing Java applications (on Puhti) diff --git a/_hands-on/installing/installing_hands-on_mcl.md b/part-2/installing/mcl.md similarity index 97% rename from _hands-on/installing/installing_hands-on_mcl.md rename to part-2/installing/mcl.md index 86688e64..e8d0208f 100644 --- a/_hands-on/installing/installing_hands-on_mcl.md +++ b/part-2/installing/mcl.md @@ -1,6 +1,12 @@ --- -topic: installing -title: Tutorial - Installing a simple C code from source +layout: default +title: Installing a simple C code from source +parent: 8. Installing own software +grand_parent: Part 2 +nav_order: 4 +has_children: false +has_toc: false +permalink: /hands-on/installing/installing_hands-on_mcl.html --- # Installing a simple C code from source diff --git a/_hands-on/installing/installing_hands-on_perl.md b/part-2/installing/perl.md similarity index 94% rename from _hands-on/installing/installing_hands-on_perl.md rename to part-2/installing/perl.md index e15ad8ab..05161e27 100644 --- a/_hands-on/installing/installing_hands-on_perl.md +++ b/part-2/installing/perl.md @@ -1,6 +1,12 @@ --- -topic: installing -title: Tutorial - Installing Perl applictions and libraries +layout: default +title: Installing Perl applications and libraries +parent: 8. Installing own software +grand_parent: Part 2 +nav_order: 5 +has_children: false +has_toc: false +permalink: /hands-on/installing/installing_hands-on_perl.html --- # Perl diff --git a/_hands-on/installing/installing_hands-on_python.md b/part-2/installing/python.md similarity index 95% rename from _hands-on/installing/installing_hands-on_python.md rename to part-2/installing/python.md index 8661e50f..143c6186 100644 --- a/_hands-on/installing/installing_hands-on_python.md +++ b/part-2/installing/python.md @@ -1,6 +1,12 @@ --- -topic: installing -title: Tutorial - Installing Python applications and libraries (essential) +layout: default +title: Installing Python packages and environments +parent: 8. Installing own software +grand_parent: Part 2 +nav_order: 2 +has_children: false +has_toc: false +permalink: /hands-on/installing/installing_hands-on_python.html --- # Python diff --git a/_hands-on/installing/installing_hands-on_r.md b/part-2/installing/r.md similarity index 93% rename from _hands-on/installing/installing_hands-on_r.md rename to part-2/installing/r.md index 5dba6e7a..3990969f 100644 --- a/_hands-on/installing/installing_hands-on_r.md +++ b/part-2/installing/r.md @@ -1,6 +1,12 @@ --- -topic: installing -title: Tutorial - Installing R applications and libraries +layout: default +title: Installing R application and libraries +parent: 8. Installing own software +grand_parent: Part 2 +nav_order: 3 +has_children: false +has_toc: false +permalink: /hands-on/installing/installing_hands-on_r.html --- # R on Puhti and package installations diff --git a/_hands-on/throughput/gaussian_hq.md b/part-2/workflows/gaussian_hq.md similarity index 97% rename from _hands-on/throughput/gaussian_hq.md rename to part-2/workflows/gaussian_hq.md index 7d9ed68f..4593f04d 100644 --- a/_hands-on/throughput/gaussian_hq.md +++ b/part-2/workflows/gaussian_hq.md @@ -1,6 +1,12 @@ --- -topic: throughput -title: Advanced tutorial - Gaussian with sbatch-hq +layout: default +title: Running Gaussian with sbatch-hq +parent: 10. How to speed up jobs +grand_parent: Part 2 +nav_order: 3 +has_children: false +has_toc: false +permalink: /hands-on/throughput/gaussian_hq.html --- # Using HyperQueue for farming Gaussian jobs on Puhti diff --git a/_hands-on/throughput/hyperqueue.md b/part-2/workflows/hyperqueue.md similarity index 96% rename from _hands-on/throughput/hyperqueue.md rename to part-2/workflows/hyperqueue.md index 42109514..a1cb7e19 100644 --- a/_hands-on/throughput/hyperqueue.md +++ b/part-2/workflows/hyperqueue.md @@ -1,6 +1,12 @@ --- -topic: throughput -title: Tutorial - Processing many files with HyperQueue +layout: default +title: Processing many files with HyperQueue +parent: 10. How to speed up jobs +grand_parent: Part 2 +nav_order: 2 +has_children: false +has_toc: false +permalink: /hands-on/throughput/hyperqueue.html --- # Using HyperQueue and local disk to process many small files efficiently diff --git a/part-2/workflows/index.md b/part-2/workflows/index.md new file mode 100644 index 00000000..b20ddf75 --- /dev/null +++ b/part-2/workflows/index.md @@ -0,0 +1,18 @@ +--- +layout: default +title: 10. How to speed up jobs +parent: Part 2 +nav_order: 3 +has_children: true +has_toc: false +--- + +# 10. How to speed up jobs + +## [10.1 Slides](https://a3s.fi/CSC_training/10_speed_up_jobs.html) + +## 10.2 Tutorials and exercises + +1. [Tutorial - Performing a simple scaling test]({{ site.baseurl }}{% link part-2/workflows/scaling.md %}) +2. [Tutorial - Processing many files with HyperQueue]({{ site.baseurl }}{% link part-2/workflows/hyperqueue.md %}) +3. [Advanced tutorial - Running Gaussian with `sbatch-hq`]({{ site.baseurl }}{% link part-2/workflows/gaussian_hq.md %}) diff --git a/_hands-on/throughput/scaling.md b/part-2/workflows/scaling.md similarity index 94% rename from _hands-on/throughput/scaling.md rename to part-2/workflows/scaling.md index d0c651ff..179ecb1d 100644 --- a/_hands-on/throughput/scaling.md +++ b/part-2/workflows/scaling.md @@ -1,6 +1,12 @@ --- -topic: throughput -title: Tutorial - Performing a simple scaling test (essential) +layout: default +title: Performing a simple scaling test +parent: 10. How to speed up jobs +grand_parent: Part 2 +nav_order: 1 +has_children: false +has_toc: false +permalink: /hands-on/throughput/scaling.html --- # Performing a simple scaling test