Description
Running cargo codspeed run
currently does not really provide any useful output. By "useful output", I mean that it should present me with some kind of performance number after running a benchmark.
Currently, the output on an "unsupported" platform looks like this:
Collected 1 benchmark suite(s) to run
Running codecov-rs pyreport
Harness: codspeed-criterion-compat v2.7.2
NOTICE: codspeed is enabled, but no performance measurement will be made since it's running in an unknown environment.
Checked: core/benches/pyreport.rs::benches::simple_report_json::simple_report_json
Checked: core/benches/pyreport.rs::benches::complex_report_json::complex_report_json
Checked: core/benches/pyreport.rs::benches::simple_chunks::simple_chunks
Checked: core/benches/pyreport.rs::benches::complex_chunks::complex_chunks
Done running pyreport
Finished running 1 benchmark suite(s)
I believe this is running the benchmarks in "check" mode, where it just runs a single iteration of the benchmark function.
Even if this is running in an unknown environment, and no proper performance measurement will be made as it is pointing out, it could at the very least capture the timing of a single iteration.
Similarly, running in a supported CI environment also only provides this output (from https://github.com/codecov/codecov-rs/actions/runs/11922994330/job/33230417322):
Collected 1 benchmark suite(s) to run
Running codecov-rs pyreport
Harness: codspeed-criterion-compat v2.7.2
Measured: core/benches/pyreport.rs::benches::simple_report_json::simple_report_json
Measured: core/benches/pyreport.rs::benches::complex_report_json::complex_report_json
Measured: core/benches/pyreport.rs::benches::simple_chunks::simple_chunks
Measured: core/benches/pyreport.rs::benches::complex_chunks::complex_chunks
Done running pyreport
Finished running 1 benchmark suite(s)
Similarly, I would expect it to output the measured time, which should be the same numbers which then appear in the codspeed dashboard (for example https://codspeed.io/codecov/codecov-rs/runs/673d14fe118024f3ad423c12)