Skip to content

Commit

Permalink
Apply suggestions and adress comments from code review Rene
Browse files Browse the repository at this point in the history
Co-authored-by: Rene Gassmoeller <[email protected]>
  • Loading branch information
MFraters and gassmoeller committed Jun 11, 2024
1 parent eaa31b5 commit 80293d5
Showing 1 changed file with 7 additions and 5 deletions.
12 changes: 7 additions & 5 deletions doc/sphinx/user/extending/benchmarking-run-time.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
# Benchmarking run time

When changing code it can sometimes be useful to if and how much your code changes make the code faster or slower. To test this, one could compile and run aspect-release compiled from the main branch and from the branch with the changed code (e.g. feature branch) and time both. The problems is that is is not very reliable, given that your computer might be doing something on the background while you are running one, but not while you are running the other.
When changing ASPECT's code it can be useful to check if and how much the new changes affect the run time of ASPECT. To test this, one could compile ASPECT in release mode on the main branch and on the branch with the changed code (e.g. feature branch), run both models and time both runs. The problems is that this is not very reliable, because your computer might be doing something in the background on one but not both of the runs.

```{note}
When running timing benchmarks, always run in release mode.
When running timing benchmarks, always make sure to compile in release mode. Debug mode is for testing model implementations, but the change to release mode affects the performance of different parts of the code differently. Useful timing information can only be generated in release mode (see {ref}`sec:run-aspect:debug-mode`).
```

One solution is to this problem is to run both executables after each other in a random order. This is what the program [cbdr](https://crates.io/crates/cbdr) can do for you. It can then also analyze and report back the result.
One solution to this problem is to run both executables after each other in a random order. This is what the program [cbdr](https://crates.io/crates/cbdr) can do for you. It can also analyze and report back the result.

This program can be installed with [cargo](https://crates.io/) (see Install Cargo button), with `cargo install cbdr`. To generate the run data, program is run with `cbdr sample --timeout=900s main:/path/to/build-main/aspect-release /path/to/test.prm features:/path/to/build-feature/aspect-release /path/to/test.prm > results.csv`. This first run runs both the main and feature branch executable to "warm up" and then it keeps randomly selection one of the two executables and write the timings out to `results.csv` until the timer reaches 900 seconds.
This program can be installed with [cargo](https://crates.io/) (see Install Cargo button), with `cargo install cbdr`. To generate the run data, the program is run with `cbdr sample --timeout=900s main:/path/to/build-main/aspect-release /path/to/test.prm features:/path/to/build-feature/aspect-release /path/to/test.prm > results.csv`. This command runs both the main and feature branch executable to "warm up" and then it keeps randomly selection one of the two executables and write the timings out to `results.csv` until the timer reaches 900 seconds.

The `results.csv` can then be summarized with `cbdr analyze <results.csv`, which produces the following output:

Expand All @@ -20,7 +20,9 @@ wall_time 6.556 ± 0.045 6.498 ± 0.046 [ -1.3% .. -0.5%]
samples 66 74
```

You can ignore the sys_time in the output above. I just shows how much time the programs spends on kernel calls. The user time is how much the program actually spending on the cpu and the wall time is how much time the program takes if you measure it by have a stop-watch next to your computer. In this case it is nearly the same, but if you run it with many processors, the user/cpu time might be much larger then the wall time.
You can ignore the sys_time in the output above. It just shows how much time the programs spends on operating system kernel calls. The user time is how much the program actually spends on your processor and the wall time is how much time the program takes if you measure it by having a stop-watch next to your computer. In this case it is nearly the same, but if you run it with many processors, the user time might be much larger then the wall time.

In this case the feature branch is, measure by th wall time, between 1.3% and 0.5% faster than the main branch with a 99.9% convidence iterval. This means that in this case we can have reasonable convidence that the feature branch is a bit faster then the main branch.

The `cbdr` program can also create a graphical output in a vega-lite format through the command `cbdr plot <results.csv > results.vl`. This can then be converted to a png through [vl-convert](https://crates.io/crates/vl-convert) with the command `vl-convert vl2png --input results.vl --output results.png`.

Expand Down

0 comments on commit 80293d5

Please sign in to comment.