Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide useful output from cargo codspeed run #58

Open
Swatinem opened this issue Nov 27, 2024 · 0 comments
Open

Provide useful output from cargo codspeed run #58

Swatinem opened this issue Nov 27, 2024 · 0 comments

Comments

@Swatinem
Copy link

Running cargo codspeed run currently does not really provide any useful output. By "useful output", I mean that it should present me with some kind of performance number after running a benchmark.

Currently, the output on an "unsupported" platform looks like this:

   Collected 1 benchmark suite(s) to run
     Running codecov-rs pyreport
Harness: codspeed-criterion-compat v2.7.2
NOTICE: codspeed is enabled, but no performance measurement will be made since it's running in an unknown environment.
Checked: core/benches/pyreport.rs::benches::simple_report_json::simple_report_json
Checked: core/benches/pyreport.rs::benches::complex_report_json::complex_report_json
Checked: core/benches/pyreport.rs::benches::simple_chunks::simple_chunks
Checked: core/benches/pyreport.rs::benches::complex_chunks::complex_chunks
        Done running pyreport
    Finished running 1 benchmark suite(s)

I believe this is running the benchmarks in "check" mode, where it just runs a single iteration of the benchmark function.
Even if this is running in an unknown environment, and no proper performance measurement will be made as it is pointing out, it could at the very least capture the timing of a single iteration.

Similarly, running in a supported CI environment also only provides this output (from https://github.com/codecov/codecov-rs/actions/runs/11922994330/job/33230417322):

     Collected 1 benchmark suite(s) to run
       Running codecov-rs pyreport
  Harness: codspeed-criterion-compat v2.7.2
  Measured: core/benches/pyreport.rs::benches::simple_report_json::simple_report_json
  Measured: core/benches/pyreport.rs::benches::complex_report_json::complex_report_json
  Measured: core/benches/pyreport.rs::benches::simple_chunks::simple_chunks
  Measured: core/benches/pyreport.rs::benches::complex_chunks::complex_chunks
          Done running pyreport
      Finished running 1 benchmark suite(s)

Similarly, I would expect it to output the measured time, which should be the same numbers which then appear in the codspeed dashboard (for example https://codspeed.io/codecov/codecov-rs/runs/673d14fe118024f3ad423c12)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant