Skip to content

Commit

Permalink
Add Measurement and Scoring details to README.md (#467)
Browse files Browse the repository at this point in the history
Document in more detail how scoring and workload measurement works:
- Add example timeline to explain which parts contribute to the score
- Add basic formulas on how the test steps are combined into a final score
  • Loading branch information
camillobruni authored Jan 8, 2025
1 parent faf9fff commit 4dfa8fa
Show file tree
Hide file tree
Showing 3 changed files with 308 additions and 0 deletions.
22 changes: 22 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,25 @@ benchmark, actual users should benefit. In order to achieve this, it should:
- Be accessible to the public and useful to browser engineers. It should run in every modern
browser by visiting a normal web page. It should run relatively quickly, while providing
enough test coverage to be reflective of the real-world Web.

# Score and Test Measurement

Each test can contain several steps who contributed to the test duration. Note that the prepare step of a test is _unmeasured_ and thus does not contribute to the score. All following test
steps are **measured** and summed up in the test step time.

<img src="./resources/measurement-timeline.svg" />

Each step consists of the following phases:

<dl>
<dt>Sync:</dt>
<dd>Time spent in synchronous JS execution.</dd>
<dt>Async:</dt>
<dd>Time spent in asynchronous JS like Promise or setTimeout tasks scheduled by the sync-step.</dd>
</dl>

## Score

The final benchmark score is calculated based of the inverse of [geomean](https://en.wikipedia.org/wiki/Geometric_mean) of all tests. We average the score over multiple iterations with the [arithmetic mean](https://en.wikipedia.org/wiki/Arithmetic_mean). By using the geomean to combine the test durations we maintain the invariant that relative improvements are favoured equally amongst all tests, even though they can have vastly different durations.

<img src="./resources/score.svg" />
88 changes: 88 additions & 0 deletions resources/measurement-timeline.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
198 changes: 198 additions & 0 deletions resources/score.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 4dfa8fa

Please sign in to comment.