Reproducing Paper Results

[framolfese]

Hi,

At this link you have published the detailed evaluation scores for each model and benchmark under different input lengths.

I have two questions about it:

1. How can we use these numbers to obtain the same results of Figure 6 in your paper?

2. For tasks like RAG, which is divided into HotpotQA etc, there are multiple files named with "k20, k220, k440" etc.

hotpotqa-dev-multikilt_1000_k20_dep3
hotpotqa-dev-multikilt_1000_k50_dep3
hotpotqa-dev-multikilt_1000_k105_dep3
hotpotqa-dev-multikilt_1000_k220_dep3
hotpotqa-dev-multikilt_1000_k440_dep3
hotpotqa-dev-multikilt_1000_k500_dep3
hotpotqa-dev-multikilt_1000_k1000_dep3

However, there are 7 of these files for evaluation with different values of $k$, but only 5 input lengths in the paper (8k, 16k, 32k, 64k and 128k). Which input length corresponds to which file?

[zhejunliux]

llm/HELMET-main/scripts/collect_results.py

    config_files = [
        "configs/recall.yaml", "configs/recall_short.yaml", 
        "configs/rag.yaml", "configs/rag_short.yaml", 
        "configs/longqa.yaml", "configs/longqa_short.yaml", 
        "configs/summ.yaml", "configs/summ_short.yaml", 
        "configs/rerank.yaml", "configs/rerank_short.yaml", 
        "configs/icl.yaml", "configs/icl_short.yaml", 
        "configs/cite.yaml", "configs/cite_short.yaml", 
        "configs/ruler.yaml", "configs/ruler_short.yaml", 
    ]

to
llm/HELMET-main/configs/rag_short.yaml

input_max_length: 8192,16384,32768,65536,8192,16384,32768,65536,8192,16384,32768,65536,8192,16384,32768,65536
datasets: kilt_nq,kilt_nq,kilt_nq,kilt_nq,kilt_triviaqa,kilt_triviaqa,kilt_triviaqa,kilt_triviaqa,kilt_hotpotqa,kilt_hotpotqa,kilt_hotpotqa,kilt_hotpotqa,kilt_popqa_3,kilt_popqa_3,kilt_popqa_3,kilt_popqa_3
generation_max_length: 20,20,20,20,20,20,20,20,20,20,20,20,20,20,20,20
test_files: data/kilt/nq-dev-multikilt_1000_k50_dep6.jsonl,data/kilt/nq-dev-multikilt_1000_k105_dep6.jsonl,data/kilt/nq-dev-multikilt_1000_k220_dep6.jsonl,data/kilt/nq-dev-multikilt_1000_k440_dep6.jsonl,data/kilt/triviaqa-dev-multikilt_1000_k50_dep6.jsonl,data/kilt/triviaqa-dev-multikilt_1000_k105_dep6.jsonl,data/kilt/triviaqa-dev-multikilt_1000_k220_dep6.jsonl,data/kilt/triviaqa-dev-multikilt_1000_k440_dep6.jsonl,data/kilt/hotpotqa-dev-multikilt_1000_k50_dep3.jsonl,data/kilt/hotpotqa-dev-multikilt_1000_k105_dep3.jsonl,data/kilt/hotpotqa-dev-multikilt_1000_k220_dep3.jsonl,data/kilt/hotpotqa-dev-multikilt_1000_k440_dep3.jsonl,data/kilt/popqa_test_1000_k50_dep6.jsonl,data/kilt/popqa_test_1000_k105_dep6.jsonl,data/kilt/popqa_test_1000_k220_dep6.jsonl,data/kilt/popqa_test_1000_k440_dep6.jsonl
demo_files: data/kilt/nq-train-multikilt_1000_k3_dep6.jsonl,data/kilt/nq-train-multikilt_1000_k3_dep6.jsonl,data/kilt/nq-train-multikilt_1000_k3_dep6.jsonl,data/kilt/nq-train-multikilt_1000_k3_dep6.jsonl,data/kilt/triviaqa-train-multikilt_1000_k3_dep6.jsonl,data/kilt/triviaqa-train-multikilt_1000_k3_dep6.jsonl,data/kilt/triviaqa-train-multikilt_1000_k3_dep6.jsonl,data/kilt/triviaqa-train-multikilt_1000_k3_dep6.jsonl,data/kilt/hotpotqa-train-multikilt_1000_k3_dep3.jsonl,data/kilt/hotpotqa-train-multikilt_1000_k3_dep3.jsonl,data/kilt/hotpotqa-train-multikilt_1000_k3_dep3.jsonl,data/kilt/hotpotqa-train-multikilt_1000_k3_dep3.jsonl,data/kilt/popqa_test_1000_k3_dep6.jsonl,data/kilt/popqa_test_1000_k3_dep6.jsonl,data/kilt/popqa_test_1000_k3_dep6.jsonl,data/kilt/popqa_test_1000_k3_dep6.jsonl
use_chat_template: false
max_test_samples: 100
shots: 2
stop_new_line: true
model_name_or_path: meta-llama/Llama-3.1-8B-Instruct

input_max_length -> 8k、16k etc

[framolfese]

Thanks. Is there a motivation on why you exclude the *-dev-multikilt_1000_k500_dep3 file from the evaluation?

[zhejunliux]

The numbers 440 and 500 are very close, so the difference doesn't seem significant. What do you think? The author also didn't test the K20 for 4K.

[framolfese]

Im my experiments I'm doing the average between the results obtained with k440 and k500 just to use them both since they are in the same ballpark, but at this point I think it makes more sense to avoid using k500 just to be compliant. Also, I'm using the k20 and k1000 because I'm also interested in testing 4k and 128k context length.

[zhejunliux]

The author tested 128k in the configs/longqa.yaml file, and the ranking provides test results from 8k to 128k.

[framolfese]

Yes sorry I was referring to the config you mentioned.

[howard-yen]

Hi @framolfese, for the different k and lengths, it's just as @zhejunliux pointed out, only five of the files are used, and the matching is as follows (which you can see from the config file):
k50 = 8K tokens
k105 = 16K tokens
k220 = 32K tokens
k440 = 64K tokens
k1000 = 128K tokens

As for why there are k20 and k500, these were for testing purposes during development of the benchmark, so they were not used at all to generate the final numbers you see in the paper. I would recommend using just k440 for 64K instead of averaging with k500 since it will be faster and comparable with the results in the paper.

Hope this helps, let me know if you have any other questions!

[framolfese]

Ok thanks a lot for the clarifications!