[EVAL] Long Horizon Execution

[akshathmangudi]

I screwed up my previous git clone, so I had to redo the changes 😅

Description:
Approach described within #1056.

Tasks:

• Initial scaffolding of /tasks/tasks/long_horizon_execution.py
• Implement a custom scorer to parse <answer> tags.
• Complete implementation of /tasks/tasks/long_horizon_execution.py
• Evaluation and Testing

STATUS: ready for review.

Current behavior:

When we run lighteval tasks inspect long_horizon_execution, the output has been shown below:

... more lines
           "'basic', 'alive', 'cream', 'dress', 'black', 'brown', 'drama', "
           "'black', 'audio', 'brown', 'album', 'cover', 'avoid', 'aware', "
           "'event', 'dream', 'clean', 'clock', 'apple', 'above', 'close', "
           "'begin', 'allow', 'album', 'draft', 'brain', 'civil', 'faith', "
           "'death', 'coach', 'below', 'doubt', 'aware', 'cover', 'final', "
           "'allow', 'avoid', 'ahead', 'cross', 'child', 'cream', 'error', "
           "'break', 'brief', 'clock', 'final', 'dance', 'award', 'every', "
           "'chief', 'could', 'dream', 'begin', 'burst', 'audio', 'album', "
           "'cross', 'doubt', 'blood', 'child', 'brand', 'brand', 'extra', "
           "'broad', 'cloud', 'check', 'after', 'chart', 'basic', 'child', "
           "'coach', 'chair', 'faith', 'earth', 'audio', 'basic', 'field', "
           "'cloud', 'draft', 'apply', 'court', 'black', 'ahead', 'burst', "
           "'crowd', 'depth', 'enemy', 'drink', 'first', 'could', 'false', "
           "'could', 'blame', 'first', 'album', 'crowd', 'first', 'broad', "
           "'extra', 'clock', 'chart', 'fiber', 'board', 'earth', 'being', "
           "'alive', 'chart', 'avoid', 'dress', 'cloud', 'clean', 'avoid', "
           "'crash', 'clean', 'arise', 'death', 'brand', 'error']\n"
           '\n'
           'Your task: Calculate the cumulative sum after each key. The first '
           'sum is just the value of the first key. The second sum is the '
           'first value plus the second value, and so on.\n'
           '\n'
           'IMPORTANT:\n'
           '- Output your answer as a single line with comma-separated values '
           'inside <answer></answer> tags\n'
           '- Do not include any other text outside the answer tags\n'
           '- Format: <answer>value1,value2,value3,...</answer>\n'
           '- Example: If the cumulative sums are [5, 8, 12], output: '
           '<answer>5,8,12</answer>\n'
           '\n'
           'Your answer:',
  'sampling_methods': [],
  'specific': None,
  'stop_sequences': (),
  'task_name': 'long_horizon_execution',
  'unconditioned_query': None,
  'use_logits': False}

[akshathmangudi]

cc: @NathanHB

[NathanHB]

looking good ! Will run locally and review today or start of next week :)
Can you share a HUggingFace Space with the samples as described here to make it easier to verify ? 🤗

[akshathmangudi]

i ran the benchmark on HF Inference's gpt-4o but a lot of the results I am seeing are quite poor. is this expected or something wrong with the prompting that I haven't looked at yet?

https://huggingface.co/spaces/akshathmangudi/lhe-gpt4o-single

[NathanHB]

Hey ! Thanks for the hard work on this, i'm testing it locally right now. I have some small nits but it's looking almost ready !

[NathanHB]

Tested on single turn, working great with the few nits I added above. However i cannot seems to make the multiturn work, can you ping when it's ready?

[akshathmangudi]

@NathanHB it should be working now, ive created a link below that tests both single and multi-turn.

https://huggingface.co/spaces/akshathmangudi/lhe-gpt

[NathanHB]

hey @akshathmangudi that's amazing !!
The link seems broken or maybe the dataset is private ? :)

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[akshathmangudi]

sorry! it was private. made it public now :)

[NathanHB]

great ! Maybe i'm mistaken but i only see single turn eval ?

[NathanHB]

hey @akshathmangudi we are planning a release thisz week and would love the tasks you started implementing to be in it. I was just wondering if you were planning on finishing those or if i could take over ? Thanks ! 🤗

[akshathmangudi]

hey @NathanHB!

sorry, been traveling all week. i'll have some space today and tomorrow, since a lot of the comments are nits and just things i accidentally overlooked (sorry for that), ill get them ready ASAP!

[akshathmangudi]

https://huggingface.co/spaces/akshathmangudi/lhe-gpt

ive updated the space to have multi-turn evaluation. please let me know if any changes have to be made 🤗

[Copilot]

Pull request overview

This PR implements the Long Horizon Execution benchmark for evaluating language models' ability to maintain state and perform cumulative operations over long sequences. The implementation follows a research paper approach with both single-turn (process all keys at once) and multi-turn (incremental key processing) evaluation modes.

Key Changes

• Added complete task implementation with support for 7 context sizes (1024-65536) and 3 turn complexities (K=1, 2, 10)
• Implemented custom answer tag parsing scorers for extracting <answer> formatted responses
• Used binary search optimization to fit maximum items within prompt length constraints

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 9 comments.

File	Description
src/lighteval/tasks/tasks/long_horizon_execution/constants.py	Defines prompt templates and configuration constants for context sizes and turn complexities
src/lighteval/tasks/tasks/long_horizon_execution/utils.py	Implements binary search logic and prompt building functions for both single and multi-turn modes
src/lighteval/tasks/tasks/long_horizon_execution/main.py	Provides single-turn task implementation with scorer and creates task configurations
src/lighteval/tasks/tasks/long_horizon_execution/multi_turn.py	Implements multi-turn evaluation with conversation state tracking and fractional accuracy scoring

Comments suppressed due to low confidence (2)

src/lighteval/tasks/tasks/long_horizon_execution/utils.py:130

• Surplus named argument for string format. An argument named 'num_keys' is provided, but it is not required by [format "You are an AI assistant. I will provide you with a dictionary and then give you keys in groups of {k}.
  Your task is to keep a running total (starting from 0) by adding the values associated with the keys I provide.
  In each turn, I'll provide {k} keys (comma-separated).
  Respond with the current running sum, enclosed in tags.

Dictionary to maintain:
{dict_str}

Ready to start!
User: {keys_str}
Assistant:"](1).

        return PROMPT_TEMPLATE_MULTI_START.format(
            dict_str=dict_str, keys_str=keys_str, k=k, num_keys=len(first_turn_keys)
        )

src/lighteval/tasks/tasks/long_horizon_execution/utils.py:194

• Surplus named argument for string format. An argument named 'num_keys' is provided, but it is not required by [format "You are an AI assistant. I will provide you with a dictionary and then give you keys in groups of {k}.
  Your task is to keep a running total (starting from 0) by adding the values associated with the keys I provide.
  In each turn, I'll provide {k} keys (comma-separated).
  Respond with the current running sum, enclosed in tags.

Dictionary to maintain:
{dict_str}

Ready to start!
User: {keys_str}
Assistant:"](1).

    initial_prompt = PROMPT_TEMPLATE_MULTI_START.format(
        dict_str=dict_str, keys_str=first_turn_keys_str, k=k, num_keys=len(turn_chunks[0])
    )

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[akshathmangudi]

it's seems there are few valid nits that copilot has addressed, will be fixing them in a few hours

[akshathmangudi]

hey @NathanHB, addressed almost all the comments and verified that the benchmark runs. let me know if there's anything else to address :)