Skip to content

Improving the Benchmark Suite #5006

Description

@sgillespie

The benchmark suite is a highly flexible performance benchmarking framework, centered
around Shake build targets. However, it has high cognitive load for HLS non-experts. In
particular, it:

  • Has multiple interconnected projects: bench, ghcide-bench and shake-bench.
  • Constructs an entire Shake build system
  • Outputs large hierarchical CSV
  • Has high levels of abstraction

That said, I propose the following changes:

  • Consolidate bench, ghcide-bench and shake-bench into a single bench project
  • Completely remove shake
  • Replace orchestration/statistical processing with tasty-bench, if reasonably fits
  • Use familiar benchmarking terminology:
    • Examples -> Projects or Harnesses
    • Experiments -> Benchmarks
    • Configurations -> separate into two distinct concepts:
      • Test run configuration
      • HLS plugin set

Metadata

Metadata

Assignees

No one assigned

    Labels

    status: in discussionNot actionable, because discussion is still ongoing or there's no decision yet

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions