Skip to content

Commit 4de376c

Browse files
IzaakwltnIzaakwltn
committed
Adding coalton-based benchmark system
1 parent 19965eb commit 4de376c

File tree

21 files changed

+1733
-409
lines changed

21 files changed

+1733
-409
lines changed

‎Makefile

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -45,8 +45,8 @@ web-docs:
4545
bench:
4646
COALTON_ENV=release sbcl --noinform \
4747
--non-interactive \
48-
--eval "(ql:quickload :coalton/benchmarks :silent t)" \
49-
--eval "(sb-ext::without-gcing (coalton-benchmarks:run-benchmarks))"
48+
--eval "(ql:quickload :coalton/benchmarks :silent t)" \
49+
--eval "(coalton-benchmarks:run-coalton-benchmarks)"
5050

5151
.PHONY: parser-coverage
5252
parser-coverage:

‎benchmarking/README.md

Lines changed: 179 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,179 @@
1+
# Run the Coalton benchmark suite:
2+
3+
`(ql:quickload :coalton/benchmarks)` or `(asdf:load-system :coalton/benchmarks)`
4+
5+
`(in-package #:coalton-benchmarks)`
6+
7+
`(run-coalton-benchmarks)`
8+
9+
# Coalton benchmark development
10+
11+
Benchmarks can be written in any Coalton project, as long as the package imports or nicknames `#:coalton-benchmarking`.
12+
13+
## Benchmark Settings
14+
15+
### Verbose
16+
Coalton benchmarking prints to the repl by default.
17+
18+
This setting can be turned off with:
19+
20+
```
21+
(cell:write! *coalton-verbose-benchmarking* False)
22+
```
23+
24+
### Printing width
25+
Coalton benchmarks print to the repl at 90 characters wide by default.
26+
27+
This can be changed using:
28+
29+
```
30+
(cell:write! *benchmark-width* <UFix>)
31+
```
32+
33+
### Print time in cientific notation
34+
By default, times are printed using scientific notation. This can be turned off using:
35+
36+
```
37+
(ell:write! *coalton-benchmark-sci-notation* False)
38+
```
39+
40+
## Defining benchmarks:
41+
42+
Benchmarks can be defined in any Coalton package (that imports or nicknames `#:coalton-benchmarking`):
43+
44+
```
45+
;; Defining a Coalton benchmark
46+
(define-benchmark stak 1000 ; samples
47+
(fn ()
48+
(stak 18 12 6)
49+
Unit))
50+
51+
;; Defining a Lisp Benchmark
52+
(define-benchmark lisp-stak 1000
53+
(fn ()
54+
(lisp Unit ()
55+
(lisp-stak 18 12 6)
56+
Unit)))
57+
```
58+
59+
Parameterized benchmarks allow for multiple inputs to the same function:
60+
61+
```
62+
(define-parameterized-benchmark rec-fib 1000
63+
(fn (x)
64+
(fib x)
65+
Unit)
66+
(seq:make 10 15 20 25))
67+
68+
```
69+
Using `:detect-convergence?` will add standard deviation information to your run, as well as dynamic sample scaling.
70+
71+
### Dynamic sample scaling
72+
73+
74+
With `:detect-convergence?` selected, benchmarks will run one sample at a time, collecting live standard deviation information for timings:
75+
76+
```
77+
(define-parameterized-benchmark rec-fib-generic 1000
78+
(fn (x)
79+
(fib-generic-wrapped x)
80+
Unit)
81+
(seq:make 10 15 20 25)
82+
:detect-convergence? cl:t)
83+
84+
```
85+
86+
When running this mode, the `samples` input is interpreted as the minimum samples, and the benchmark will continue running until the standard deviation of time converges. This ensures the minimum amount of samples before a reliable average.
87+
88+
This mode is most effective on benchmarks with significant individual sample times.
89+
90+
## Running individual benchmarks
91+
92+
Individual benchmarks can be run with `#'run-benchmark`, as long as the benchmark is defined in the current package.
93+
94+
`#'run-benchmark` returns a `BenchmarkResults` object.
95+
96+
```
97+
COALTON-BENCHMARKS/FIBONACCI> (coalton (run-benchmark "rec-fib0"))
98+
┌─────────────────────────────────────────────────────────────────────────────────────────┐
99+
│ Benchmark rec-fib0 │
100+
├─────────────────────────────────────────────────────────────────────────────────────────┤
101+
│ System: ARM64 OS-MACOSX SBCL2.2.4-WIP │
102+
├─────────────────────────────────────────────────────────────────────────────────────────┤
103+
│ Coalton development mode without heuristic inlining │
104+
├──────────────┬──────────────┬──────────────┬──────────────┬──────────────┬──────────────┤
105+
│ Benchmark │ Time (ms) │Avg Time (ms) │ Time std dev │ Space (B) │ # Samples │
106+
├──────────────┼──────────────┼──────────────┼──────────────┼──────────────┼──────────────┤
107+
│ REC-FIB0 │ 2 │ 0 │ n/a │ 0 │ 1000 │
108+
└──────────────┴──────────────┴──────────────┴──────────────┴──────────────┴──────────────┘
109+
110+
#.(BENCHMARKRESULTS REC-FIB0 1000 2371 2 #.NONE #.(SOME 0))
111+
```
112+
113+
## Running package benchmarks
114+
115+
Package benchmarks can be run with `#'run-package-benchmarks`.
116+
117+
`#'run-package-benchmarks` returns a `PackageBenchmarkResults` object.
118+
119+
Local packages can be run with `#'run-benchmarks`.
120+
121+
```
122+
COALTON-BENCHMARKS/FIBONACCI> (coalton (run-benchmarks))
123+
┌─────────────────────────────────────────────────────────────────────────────────────────┐
124+
│ Package 'COALTON-BENCHMARKS/FIBONACCI' │
125+
├─────────────────────────────────────────────────────────────────────────────────────────┤
126+
│ System: ARM64 OS-MACOSX SBCL2.2.4-WIP │
127+
├─────────────────────────────────────────────────────────────────────────────────────────┤
128+
│ Coalton development mode without heuristic inlining │
129+
├──────────────┬──────────────┬──────────────┬──────────────┬──────────────┬──────────────┤
130+
│ Benchmark │ Time (ms) │Avg Time (ms) │ Time std dev │ Space (B) │ # Samples │
131+
├──────────────┼──────────────┼──────────────┼──────────────┼──────────────┼──────────────┤
132+
│ REC-FIB0 │ 2 │ 0 │ n/a │ 0 │ 1000 │
133+
├──────────────┼──────────────┼──────────────┼──────────────┼──────────────┼──────────────┤
134+
│ REC-FIB1 │ 20 │ 0 │ n/a │ 0 │ 1000 │
135+
├──────────────┼──────────────┼──────────────┼──────────────┼──────────────┼──────────────┤
136+
│ REC-FIB2 │ 216 │ 0 │ n/a │ 0 │ 1000 │
137+
├──────────────┼──────────────┼──────────────┼──────────────┼──────────────┼──────────────┤
138+
│ REC-FIB3 │ 2370 │ 2 │ n/a │ 0 │ 1000 │
139+
├──────────────┼──────────────┼──────────────┼──────────────┼──────────────┼──────────────┤
140+
│ REC-FIB-GENE │ 17 │ 0 │ 0.0055017565 │ 0 │ 1002 │
141+
├──────────────┼──────────────┼──────────────┼──────────────┼──────────────┼──────────────┤
142+
│ REC-FIB-GENE │ 95 │ 0 │ 0.0053594956 │ 0 │ 1002 │
143+
├──────────────┼──────────────┼──────────────┼──────────────┼──────────────┼──────────────┤
144+
│ REC-FIB-GENE │ 912 │ 1 │ 0.0226823009 │ 0 │ 1002 │
145+
├──────────────┼──────────────┼──────────────┼──────────────┼──────────────┼──────────────┤
146+
│ REC-FIB-GENE │ 10270 │ 10 │ 0.3188679543 │ 0 │ 1014 │
147+
├──────────────┼──────────────┼──────────────┼──────────────┼──────────────┼──────────────┤
148+
│ REC-FIB-LISP │ 105 │ 0 │ n/a │ 0 │ 1000 │
149+
├──────────────┼──────────────┼──────────────┼──────────────┼──────────────┼──────────────┤
150+
│ REC-FIB-MONO │ 306 │ 0 │ n/a │ 0 │ 1000 │
151+
└───────────────────────┴─────────────────────┴─────────────────────┴─────────────────────┘
152+
#.(PACKAGEBENCHMARKRESULTS "COALTON-BENCHMARKS/FIBONACCI" #.(BENCHMARKSYSTEM "ARM64" "OS-MACOSX" "SBCL" "2.2.4-WIP" COMMON-LISP:NIL COMMON-LISP:NIL) #(#.(BENCHMARKRESULTS REC-FIB0 1000 2476 2 #.NONE #.(SOME 0))
153+
#.(BENCHMARKRESULTS REC-FIB1 1000 20065 20 #.NONE #.(SOME 0))
154+
#.(BENCHMARKRESULTS REC-FIB2 1000 215897 216 #.NONE #.(SOME 0))
155+
#.(BENCHMARKRESULTS REC-FIB3 1000 2370219 2370 #.NONE #.(SOME 0))
156+
#.(BENCHMARKRESULTS REC-FIB-GENERIC0 1002 16914 17 #.(SOME 5.501756508182156d0) #.(SOME 0))
157+
#.(BENCHMARKRESULTS REC-FIB-GENERIC1 1002 94676 94 #.(SOME 5.359495667149465d0) #.(SOME 0))
158+
#.(BENCHMARKRESULTS REC-FIB-GENERIC2 1002 911697 910 #.(SOME 22.68230092439423d0) #.(SOME 0))
159+
#.(BENCHMARKRESULTS REC-FIB-GENERIC3 1014 10270171 10128 #.(SOME 318.8679543261109d0) #.(SOME 0))
160+
#.(BENCHMARKRESULTS REC-FIB-LISP 1000 104998 105 #.NONE #.(SOME 0))
161+
#.(BENCHMARKRESULTS REC-FIB-MONO 1000 306261 306 #.NONE #.(SOME 0))))
162+
```
163+
164+
`#:run-benchmarks` runs the current package's benchmarks.
165+
166+
## Reexporting package benchmarks
167+
Package benchmarks can be manually run from other packages simply by defining a helper function, as in `#:coalton-benchmarks/gabriel`.
168+
169+
```
170+
(coalton-toplevel
171+
172+
(define (run-gabriel-benchmarks)
173+
(run-package-benchmarks "coalton-benchmarks/gabriel/tak")
174+
(run-package-benchmarks "coalton-benchmarks/gabriel/takr")
175+
(run-package-benchmarks "coalton-benchmarks/gabriel/stak")
176+
(run-package-benchmarks "coalton-benchmarks/gabriel/takl")))
177+
```
178+
179+
This is useful for package-per-file projects.

0 commit comments

Comments
 (0)