Skip to content

Provide useful output from cargo codspeed run #58

Open
@Swatinem

Description

@Swatinem

Running cargo codspeed run currently does not really provide any useful output. By "useful output", I mean that it should present me with some kind of performance number after running a benchmark.

Currently, the output on an "unsupported" platform looks like this:

   Collected 1 benchmark suite(s) to run
     Running codecov-rs pyreport
Harness: codspeed-criterion-compat v2.7.2
NOTICE: codspeed is enabled, but no performance measurement will be made since it's running in an unknown environment.
Checked: core/benches/pyreport.rs::benches::simple_report_json::simple_report_json
Checked: core/benches/pyreport.rs::benches::complex_report_json::complex_report_json
Checked: core/benches/pyreport.rs::benches::simple_chunks::simple_chunks
Checked: core/benches/pyreport.rs::benches::complex_chunks::complex_chunks
        Done running pyreport
    Finished running 1 benchmark suite(s)

I believe this is running the benchmarks in "check" mode, where it just runs a single iteration of the benchmark function.
Even if this is running in an unknown environment, and no proper performance measurement will be made as it is pointing out, it could at the very least capture the timing of a single iteration.

Similarly, running in a supported CI environment also only provides this output (from https://github.com/codecov/codecov-rs/actions/runs/11922994330/job/33230417322):

     Collected 1 benchmark suite(s) to run
       Running codecov-rs pyreport
  Harness: codspeed-criterion-compat v2.7.2
  Measured: core/benches/pyreport.rs::benches::simple_report_json::simple_report_json
  Measured: core/benches/pyreport.rs::benches::complex_report_json::complex_report_json
  Measured: core/benches/pyreport.rs::benches::simple_chunks::simple_chunks
  Measured: core/benches/pyreport.rs::benches::complex_chunks::complex_chunks
          Done running pyreport
      Finished running 1 benchmark suite(s)

Similarly, I would expect it to output the measured time, which should be the same numbers which then appear in the codspeed dashboard (for example https://codspeed.io/codecov/codecov-rs/runs/673d14fe118024f3ad423c12)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions