Search results

Improving input token throughput?

Status: Open.

#1063 In huggingface/lighteval;

· carlesoctav opened on Nov 17, 2025

[EVAL] Long Horizon Execution

good first issue

help wanted

new-task

Status: Open.

#1056 In huggingface/lighteval;

· NathanHB opened on Nov 13, 2025

[BUG] git clone issues due to filenames with special characters

bug

Status: Open.

#1055 In huggingface/lighteval;

· amosyou opened on Nov 13, 2025

[FT] Prompt Level Caching

Status: Open.

#1053 In huggingface/lighteval;

· f14-bertolotti opened on Nov 12, 2025

[BUG] Got Zero em on MMLU using Deepseek distilled Qwen models

bug

Status: Open.

#1047 In huggingface/lighteval;

· Zoeyyao27 opened on Nov 11, 2025

[BUG] [eval] --max-samples 10 and then without, you get the cached data from the previous run.

bug

Status: Open.

#1040 In huggingface/lighteval;

· NathanHB opened on Nov 6, 2025

[EVAL] Add kyrgyzLLM benchmark

good first issue

new-task

Status: Open.

#1036 In huggingface/lighteval;

· golden-ratio opened on Nov 4, 2025

[BUG] Python API doesn't seem to work with lighteval|gsm8k|<few-shot-number>

bug

Status: Open.

#1029 In huggingface/lighteval;

· Saravana-Ace opened on Oct 24, 2025

How to evaluate MMLU-Pro

Status: Open.

#1028 In huggingface/lighteval;

· qhz991029 opened on Oct 24, 2025

[BUG] Cache management failed for tasks using sample_params to update function attrs

bug

Status: Open.

#1026 In huggingface/lighteval;

· colinzuo opened on Oct 22, 2025

[BUG] LightevalTaskConfig doesn't support comparison, but Cache management uses sorted on it

bug

Status: Open.

#1024 In huggingface/lighteval;

· colinzuo opened on Oct 21, 2025

[BUG] Cache management failed for tasks using MetricGrouping

bug

Status: Open.

#1023 In huggingface/lighteval;

· colinzuo opened on Oct 21, 2025