Deployed 38adb6a with MkDocs version: 1.6.1

index.html

@@ -676,6 +676,14 @@ <h2 id="4-submit-the-workflow-to-scheduler">4. Submit the workflow to Scheduler<
 <p class="admonition-title">Launch it with the command <code>source submit.sh</code></p>
 <p>What does it do : </p>
 <p><center><a class="glightbox" href="images/workflow.png" data-type="image" data-width="auto" data-height="auto" data-desc-position="bottom"><img alt="image" src="images/workflow.png" width="500" /></a></center></p>
+<p><br></p>
+<details class="video">
+<summary><code>source submit.sh</code></summary>
+<p><video width="100%" controls>
+  <source src="https://github.com/user-attachments/assets/be6820fa-93ff-485e-b786-b826046e8e08" type="video/mp4">
+  Your browser does not support the video tag.
+</video></p>
+</details>
 <details class="clipboard-question">
 <summary>Reason for using <code>source</code> and not <code>bash</code></summary>
 <ul>

search/search_index.json

@@ -1 +1 @@
-{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Home","text":"<p>"},{"location":"#apsim-hpc","title":"APSIM-HPC","text":"<p>Deploy APSIM (Agricultural Production Systems sIMulator - https://www.apsim.info/) on high performance computing clusters.</p> <p> </p>"},{"location":"#1-apptainer-build","title":"1. Apptainer build","text":"<p>Building the APSIM container</p> <pre><code>git clone https://github.com/nesi/APSIM-HPC.git\ncd APSIM-HPC/01-apptainer-build\n./updateversion-and-build.sh\n</code></pre> <code>./updateversion-and-build.sh</code> <p>    Your browser does not support the video tag. </p> <p></p>"},{"location":"#2-setup-the-working-directory","title":"2. Setup the working directory","text":"<p>Structure/Content of the working directory</p> <p>APSIM requires Weather files ( .met), Template to create the config files, CSV files with soil,file,database names and soil library .apsimx files to be on the same working directory.</p> <pre><code>\u251c\u2500\u2500 1.met\n\u251c\u2500\u2500 2.met\n\u251c\u2500\u2500 3.met\n\u251c\u2500\u2500 another_soilapsim_library.apsimx\n\u251c\u2500\u2500 ExampleConfigTemplate.txt\n\u251c\u2500\u2500 fileanddbnames.csv\n\u251c\u2500\u2500 my_master_soilapsim_library.apsimx\n\u2514\u2500\u2500 soilnames.csv\n</code></pre> <p></p>"},{"location":"#3-update-workflow-files-for-current-simulation","title":"3. Update Workflow files for current simulation","text":"<p>Please make sure to edit the files within the cloned repository without changing the repository directory structure or filenames. Final submission script relies on the existing structure and names to populate the working directory with runtime scripts.</p>"},{"location":"#03-generate-config-filesgenerate_apsim_configsr","title":"<code>03-generate-config-files/generate_apsim_configs.R</code>","text":"<ul> <li>This script generates the config.txt files ( one file per soil sample per weather file)</li> <li>It will be executed interactively than submitting as a job to scheduler ( relatively quick -  generating 10,000 files will be done within 75 seconds or so)</li> <li>Last step of this script is to \"split\" those config files to multiple Sets (Default is 3 Sets)<ul> <li>What is the purpose of this split ?<ul> <li>This a pre-configuration for <code>08-snakemake/Snakefile_1</code> step. Latter process the config files and create the corresponding .apsimx files ( plus the .db placeholder files). It can not be done in parallel due to https://github.com/nesi/APSIM-HPC/issues/31. </li> <li>Since the processing time per file is ~25 seconds, having all of the configfiles ( assuming there are thousands to process) in one single directory will add unreasonable runtimes. Safest ( and the easiest) soution is to split those files to multiple Sets ( processing within a set is still serial but it will be quicker as it will be a fraction of the total samples)</li> </ul> </li> </ul> </li> </ul> <p>update <code>03-generate-config-files/generate_apsim_configs.R</code></p> <p>First file to be edited is <code>03-generate-config-files/generate_apsim_configs.R</code> which will be used to generate the config.txt files</p> <ul> <li>Review line number 6, 8 and 11 </li> </ul> <pre><code>6 SoilNames &lt;- as.vector(unlist(read.csv(\"soilnames.csv\"))) ##List of soil names that must exist in the Soil Library .apsimx file (most likely supplied by experienced APSIM user)\n\n8 ConfigTemplate &lt;- \"ExampleConfigTemplate.txt\" ##Base config file\n\n11 SoilLibrary &lt;- \"my_master_soilapsim_library.apsimx\" ###Change name to the soil library you are using\n</code></pre> <ul> <li>if you would like to change the number of Sets from default <code>3</code>, change the value of <code>nSets</code> on line 87</li> </ul> <pre><code>87 SplitConfigFilesIntoSets(list.files(pattern = \"ConfigFile.txt$\"), nSets = 3)\n</code></pre>"},{"location":"#08-snakemakesnakefile_1","title":"<code>08-snakemake/Snakefile_1</code>","text":"<ul> <li>This step is scheduler based. It generates the .apsimx ( and placeholder .db files) from config files</li> <li>This is a serial process  ( as described above) which makes it the longest step in terms of runtime in the workflow ( take ~25 seconds per config file)</li> <li>Default runtime assigned to this step is 16 hours ( this is sufficient to handle a set of 2300 files)</li> </ul> <p>update <code>08-snakemake/Snakefile_1</code></p> <ul> <li>All major changes are within the <code>config</code> block<ul> <li><code>apptainer_bind</code> - We recommend binding the full filesystem. <ul> <li>For eRI, It will be <code>\"apptainer_bind\": \"/agr/scratch,/agr/persist\"</code></li> <li>For Mahuika, it's <code>\"apptainer_bind\": \"/nesi/project,/nesi/nobackup,/opt/nesi\"</code></li> </ul> </li> <li><code>\"excluded_txt_files\"</code> is the name of the template config which will excluded during the processing. Otherwise, it will create a .apsimx and a place holder .db file for itself</li> </ul> </li> </ul> <pre><code>   4 config = {\n   5     \"apptainer_bind\": \"/full/filesystem\",\n   6     \"apptainer_image\": \"/path/to/container/image/version.aimg\",\n   7     \"excluded_txt_files\": [\"ExampleConfigTemplate.txt\"],\n   8     \"max_consecutive_failures\": 10,\n   9     \"slurm_logdir\": \"slurmlogs\"\n  10 }\n</code></pre> <ul> <li>If you would like to increase the runtime, refer to line 28 </li> </ul> <pre><code>  28         time = \"16:00:00\"\n</code></pre>"},{"location":"#08-snakemakesnakefile_2","title":"<code>08-snakemake/Snakefile_2</code>","text":"<ul> <li>This step is scheduler based. It generates the .db from .apsimx files</li> <li>Processing will be done in parallel .i.e. Each .apsimx file will have it's own job. Average processing time per file at 12 cpus varies between 50 seconds to 5 minutes. We have set the default to be 10 minutes per file. Similar to above, change the <code>time =</code> varibale as needed.</li> </ul> <p>update <code>08-snakemake/Snakefile_2</code></p> <ul> <li>Simlar to above, all changes are within the <code>config</code> block<ul> <li><code>\"excluded_apsimx_files\"</code> is/are the name/s of the soil library .apsimx files which will excluded during the processing. Otherwise, it will create a .db file/s for those</li> </ul> </li> </ul> <pre><code>   5 config = {\n   6     \"apptainer_bind\": \"/full/filesystem\",\n   7     \"apptainer_image\": \"/path/to/container/image/version.aimg\",\n   8     \"excluded_apsimx_files\": [\"my_master_soilapsim_library.apsimx\", \"another_soilapsim_library.apsimx\"],\n   9     \"slurm_logdir\": \"slurmlogs\",\n  10     \"size_threshold\": 1 * 1024 * 1024  # 1MB in bytes\n  11 }\n</code></pre> <p></p>"},{"location":"#4-submit-the-workflow-to-scheduler","title":"4. Submit the workflow to Scheduler","text":"<p>Assumptions and pre-requisites</p> <ol> <li>Cloned the latest version of the repo (https://github.com/nesi/APSIM-HPC.git)</li> <li><code>03-generate-config-files/generate_apsim_configs.R</code> , <code>08-snakemake/Snakefile_1</code> and <code>08-snakemake/Snakefile_2</code> were updated with names of the template files,etc. </li> <li>Filenames and directory structure of the cloned repo were/was not altered.<ul> <li>submission script relies on the relative paths and names as it is within the Github repo.</li> <li>If you are to change paths or names, make sure to review <code>submit.sh</code> and amend to reflect the changes.  </li> </ul> </li> </ol> <p>Launch it with the command <code>source submit.sh</code></p> <p>What does it do : </p> <p></p> Reason for using <code>source</code> and not <code>bash</code> <ul> <li> <p>This is do with <code>cd /working/directory</code> command within the <code>submit.sh</code> script</p> </li> <li> <p>When you run a script using <code>bash script.sh</code>, the script is executed in a new subshell. This means:</p> <ul> <li>A new shell process is created to run the script.</li> <li>Any changes to the working directory (<code>cd</code> commands) only affect this subshell.</li> <li>Once the script finishes, the subshell terminates, and you return to your original shell with its original working directory unchanged.</li> </ul> </li> </ul>"},{"location":"1-runtime-info/","title":"Runtime data","text":"<p> Runtime for  one sample across varying number of CPUs: This is for the second phase of the workflow which creates the .db files from .apxims files (<code>08-snakemake/Snakefile_2</code>) </p>"},{"location":"1-runtime-info/#runtime-behaviour","title":"Runtime behaviour","text":"4cpu 8cpu 12cpu 16cpu 20cpu 24cpu"}]}
+{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Home","text":"<p>"},{"location":"#apsim-hpc","title":"APSIM-HPC","text":"<p>Deploy APSIM (Agricultural Production Systems sIMulator - https://www.apsim.info/) on high performance computing clusters.</p> <p> </p>"},{"location":"#1-apptainer-build","title":"1. Apptainer build","text":"<p>Building the APSIM container</p> <pre><code>git clone https://github.com/nesi/APSIM-HPC.git\ncd APSIM-HPC/01-apptainer-build\n./updateversion-and-build.sh\n</code></pre> <code>./updateversion-and-build.sh</code> <p>    Your browser does not support the video tag. </p> <p></p>"},{"location":"#2-setup-the-working-directory","title":"2. Setup the working directory","text":"<p>Structure/Content of the working directory</p> <p>APSIM requires Weather files ( .met), Template to create the config files, CSV files with soil,file,database names and soil library .apsimx files to be on the same working directory.</p> <pre><code>\u251c\u2500\u2500 1.met\n\u251c\u2500\u2500 2.met\n\u251c\u2500\u2500 3.met\n\u251c\u2500\u2500 another_soilapsim_library.apsimx\n\u251c\u2500\u2500 ExampleConfigTemplate.txt\n\u251c\u2500\u2500 fileanddbnames.csv\n\u251c\u2500\u2500 my_master_soilapsim_library.apsimx\n\u2514\u2500\u2500 soilnames.csv\n</code></pre> <p></p>"},{"location":"#3-update-workflow-files-for-current-simulation","title":"3. Update Workflow files for current simulation","text":"<p>Please make sure to edit the files within the cloned repository without changing the repository directory structure or filenames. Final submission script relies on the existing structure and names to populate the working directory with runtime scripts.</p>"},{"location":"#03-generate-config-filesgenerate_apsim_configsr","title":"<code>03-generate-config-files/generate_apsim_configs.R</code>","text":"<ul> <li>This script generates the config.txt files ( one file per soil sample per weather file)</li> <li>It will be executed interactively than submitting as a job to scheduler ( relatively quick -  generating 10,000 files will be done within 75 seconds or so)</li> <li>Last step of this script is to \"split\" those config files to multiple Sets (Default is 3 Sets)<ul> <li>What is the purpose of this split ?<ul> <li>This a pre-configuration for <code>08-snakemake/Snakefile_1</code> step. Latter process the config files and create the corresponding .apsimx files ( plus the .db placeholder files). It can not be done in parallel due to https://github.com/nesi/APSIM-HPC/issues/31. </li> <li>Since the processing time per file is ~25 seconds, having all of the configfiles ( assuming there are thousands to process) in one single directory will add unreasonable runtimes. Safest ( and the easiest) soution is to split those files to multiple Sets ( processing within a set is still serial but it will be quicker as it will be a fraction of the total samples)</li> </ul> </li> </ul> </li> </ul> <p>update <code>03-generate-config-files/generate_apsim_configs.R</code></p> <p>First file to be edited is <code>03-generate-config-files/generate_apsim_configs.R</code> which will be used to generate the config.txt files</p> <ul> <li>Review line number 6, 8 and 11 </li> </ul> <pre><code>6 SoilNames &lt;- as.vector(unlist(read.csv(\"soilnames.csv\"))) ##List of soil names that must exist in the Soil Library .apsimx file (most likely supplied by experienced APSIM user)\n\n8 ConfigTemplate &lt;- \"ExampleConfigTemplate.txt\" ##Base config file\n\n11 SoilLibrary &lt;- \"my_master_soilapsim_library.apsimx\" ###Change name to the soil library you are using\n</code></pre> <ul> <li>if you would like to change the number of Sets from default <code>3</code>, change the value of <code>nSets</code> on line 87</li> </ul> <pre><code>87 SplitConfigFilesIntoSets(list.files(pattern = \"ConfigFile.txt$\"), nSets = 3)\n</code></pre>"},{"location":"#08-snakemakesnakefile_1","title":"<code>08-snakemake/Snakefile_1</code>","text":"<ul> <li>This step is scheduler based. It generates the .apsimx ( and placeholder .db files) from config files</li> <li>This is a serial process  ( as described above) which makes it the longest step in terms of runtime in the workflow ( take ~25 seconds per config file)</li> <li>Default runtime assigned to this step is 16 hours ( this is sufficient to handle a set of 2300 files)</li> </ul> <p>update <code>08-snakemake/Snakefile_1</code></p> <ul> <li>All major changes are within the <code>config</code> block<ul> <li><code>apptainer_bind</code> - We recommend binding the full filesystem. <ul> <li>For eRI, It will be <code>\"apptainer_bind\": \"/agr/scratch,/agr/persist\"</code></li> <li>For Mahuika, it's <code>\"apptainer_bind\": \"/nesi/project,/nesi/nobackup,/opt/nesi\"</code></li> </ul> </li> <li><code>\"excluded_txt_files\"</code> is the name of the template config which will excluded during the processing. Otherwise, it will create a .apsimx and a place holder .db file for itself</li> </ul> </li> </ul> <pre><code>   4 config = {\n   5     \"apptainer_bind\": \"/full/filesystem\",\n   6     \"apptainer_image\": \"/path/to/container/image/version.aimg\",\n   7     \"excluded_txt_files\": [\"ExampleConfigTemplate.txt\"],\n   8     \"max_consecutive_failures\": 10,\n   9     \"slurm_logdir\": \"slurmlogs\"\n  10 }\n</code></pre> <ul> <li>If you would like to increase the runtime, refer to line 28 </li> </ul> <pre><code>  28         time = \"16:00:00\"\n</code></pre>"},{"location":"#08-snakemakesnakefile_2","title":"<code>08-snakemake/Snakefile_2</code>","text":"<ul> <li>This step is scheduler based. It generates the .db from .apsimx files</li> <li>Processing will be done in parallel .i.e. Each .apsimx file will have it's own job. Average processing time per file at 12 cpus varies between 50 seconds to 5 minutes. We have set the default to be 10 minutes per file. Similar to above, change the <code>time =</code> varibale as needed.</li> </ul> <p>update <code>08-snakemake/Snakefile_2</code></p> <ul> <li>Simlar to above, all changes are within the <code>config</code> block<ul> <li><code>\"excluded_apsimx_files\"</code> is/are the name/s of the soil library .apsimx files which will excluded during the processing. Otherwise, it will create a .db file/s for those</li> </ul> </li> </ul> <pre><code>   5 config = {\n   6     \"apptainer_bind\": \"/full/filesystem\",\n   7     \"apptainer_image\": \"/path/to/container/image/version.aimg\",\n   8     \"excluded_apsimx_files\": [\"my_master_soilapsim_library.apsimx\", \"another_soilapsim_library.apsimx\"],\n   9     \"slurm_logdir\": \"slurmlogs\",\n  10     \"size_threshold\": 1 * 1024 * 1024  # 1MB in bytes\n  11 }\n</code></pre> <p></p>"},{"location":"#4-submit-the-workflow-to-scheduler","title":"4. Submit the workflow to Scheduler","text":"<p>Assumptions and pre-requisites</p> <ol> <li>Cloned the latest version of the repo (https://github.com/nesi/APSIM-HPC.git)</li> <li><code>03-generate-config-files/generate_apsim_configs.R</code> , <code>08-snakemake/Snakefile_1</code> and <code>08-snakemake/Snakefile_2</code> were updated with names of the template files,etc. </li> <li>Filenames and directory structure of the cloned repo were/was not altered.<ul> <li>submission script relies on the relative paths and names as it is within the Github repo.</li> <li>If you are to change paths or names, make sure to review <code>submit.sh</code> and amend to reflect the changes.  </li> </ul> </li> </ol> <p>Launch it with the command <code>source submit.sh</code></p> <p>What does it do : </p> <p></p> <p></p> <code>source submit.sh</code> <p>    Your browser does not support the video tag. </p> Reason for using <code>source</code> and not <code>bash</code> <ul> <li> <p>This is do with <code>cd /working/directory</code> command within the <code>submit.sh</code> script</p> </li> <li> <p>When you run a script using <code>bash script.sh</code>, the script is executed in a new subshell. This means:</p> <ul> <li>A new shell process is created to run the script.</li> <li>Any changes to the working directory (<code>cd</code> commands) only affect this subshell.</li> <li>Once the script finishes, the subshell terminates, and you return to your original shell with its original working directory unchanged.</li> </ul> </li> </ul>"},{"location":"1-runtime-info/","title":"Runtime data","text":"<p> Runtime for  one sample across varying number of CPUs: This is for the second phase of the workflow which creates the .db files from .apxims files (<code>08-snakemake/Snakefile_2</code>) </p>"},{"location":"1-runtime-info/#runtime-behaviour","title":"Runtime behaviour","text":"4cpu 8cpu 12cpu 16cpu 20cpu 24cpu"}]}

stylesheets/extra.css

@@ -1114,7 +1114,7 @@
 }
 .md-typeset .video > .admonition-title,
 .md-typeset .video > summary {
-  background-color: rgba(255,255,0, 0.2);
+  background-color: rgba(202, 226, 95, 0.705);
 }
 .md-typeset .video > .admonition-title::before,
 .md-typeset .video > summary::before {