Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add airflow benchmark #3643

Merged
merged 9 commits into from
May 23, 2024
Merged

Conversation

ibraheemdev
Copy link
Member

Summary

This seems to be one of the most consistent benchmark cases we have in terms of standard deviation:

➜  hyperfine "target/profiling/main pip compile scripts/requirements/airflow.in" --runs 200
Benchmark 1: target/profiling/main pip compile scripts/requirements/airflow.in
  Time (mean ± σ):     292.6 ms ±   6.6 ms    [User: 414.1 ms, System: 194.2 ms]
  Range (min … max):   282.7 ms … 320.1 ms    200 runs

For smaller benchmarks, scispacy and dtlssocket seem to be a bit more consistent than our current jupyter benchmark, but it hasn't given us any problems so I'll leave it for now.

@zanieb zanieb added performance Potential performance improvement testing Internal testing of behavior labels May 17, 2024
@ibraheemdev
Copy link
Member Author

Hm we might have to install source distributions in a separate CI step to get CodSpeed to work.

@konstin
Copy link
Member

konstin commented May 17, 2024

Since this is a cached case benchmark, we can build all dependencies (pip build should do), upload them to github pages and use --find-links. I can't tell if the effort is justified here though.

@konstin
Copy link
Member

konstin commented May 21, 2024

In #3688 we had a massive speedup on airflow but nearly no effect on jupyter, so this would be helpful for this case.

@ibraheemdev ibraheemdev force-pushed the airflow-bench branch 10 times, most recently from a930641 to 85caa47 Compare May 21, 2024 19:08
@ibraheemdev ibraheemdev added the benchmarks Related to benchmarking label May 23, 2024
Copy link

codspeed-hq bot commented May 23, 2024

CodSpeed Performance Report

Merging #3643 will not alter performance

Comparing ibraheemdev:airflow-bench (bacd1a0) with main (8f0f4d2)

Summary

✅ 12 untouched benchmarks

🆕 1 new benchmarks

Benchmarks breakdown

Benchmark main ibraheemdev:airflow-bench Change
🆕 resolve_warm_airflow N/A 1.5 s N/A

@charliermarsh charliermarsh merged commit 0d2f3fc into astral-sh:main May 23, 2024
44 of 45 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
benchmarks Related to benchmarking performance Potential performance improvement testing Internal testing of behavior
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants