Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial attempt at a Criterion based benchmark. #17

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

ximon18
Copy link
Member

@ximon18 ximon18 commented Jan 18, 2022

See the additions to README.md for guidance on running the benchmark.

Note that it appears from the Criterion documentation that it might be advisable to use iai instead of criterion for benchmarks run in a CI environment like GitHub Actions:

If you absolutely need to pick one or the other though, Iai is probably the one to go with.

However, the documentation and capabilities of iai seem to be very limited and these benchmarks are comparatively long running and perhaps less susceptible to distortion of the results, so I'm not sure about this.

Command line output:

$ cargo criterion
   Compiling rotonda-store v0.3.0-dev (/run/media/ximon/NewHome/src/rotonda-store)
...
    Finished bench [optimized] target(s) in 4.68s
Benchmarking tree insertion/[4]: Warming up for 3.0000 s
Warning: Unable to complete 20 samples in 5.0s. You may wish to increase target time to 18.4s, or reduce sample count to 10.
tree insertion/[4]      time:   [863.52 ms 872.47 ms 882.47 ms]                             
                        thrpt:  [1.0130 Melem/s 1.0246 Melem/s 1.0352 Melem/s]
                 change:
                        time:   [-5.6231% -4.3701% -3.1893%] (p = 0.00 < 0.05)
                        thrpt:  [+3.2943% +4.5698% +5.9582%]
                        Performance has improved.
Benchmarking tree insertion/[3, 4, 5, 4]: Warming up for 3.0000 s
Warning: Unable to complete 20 samples in 5.0s. You may wish to increase target time to 30.4s, or reduce sample count to 10.
tree insertion/[3, 4, 5, 4]                                                                          
                        time:   [1.3017 s 1.3141 s 1.3306 s]
                        thrpt:  [671.83 Kelem/s 680.28 Kelem/s 686.77 Kelem/s]
                 change:
                        time:   [-0.1193% +1.0924% +2.5968%] (p = 0.12 > 0.05)
                        thrpt:  [-2.5311% -1.0806% +0.1194%]
                        No change in performance detected.
Benchmarking tree insertion/[3, 3, 3, 3, 3, 3, 3, 3, 4, 4]: Warming up for 3.0000 s
Warning: Unable to complete 20 samples in 5.0s. You may wish to increase target time to 26.9s, or reduce sample count to 10.
tree insertion/[3, 3, 3, 3, 3, 3, 3, 3, 4, 4]                                                                          
                        time:   [1.1309 s 1.2603 s 1.4559 s]
                        thrpt:  [613.99 Kelem/s 709.34 Kelem/s 790.47 Kelem/s]
                 change:
                        time:   [+9.7783% +21.993% +41.776%] (p = 0.00 < 0.05)
                        thrpt:  [-29.466% -18.028% -8.9073%]
...

HTML output:

image

image

image

@ximon18
Copy link
Member Author

ximon18 commented Jan 19, 2022

@density215: I've marked this PR as a "draft" because I'd like your feedback on whether this is the kind of thing you had in mind. Then we can discuss how to add to or alter it, e.g. automate it with GH Actions (if the results will not be unstable), exercise and report on the code in different ways, e.g. extend it to compare the single threaded store vs the multi-threaded store, etc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant