How to understand the mearning of hotpotqa-dev-multikilt_1000_k1000_dep3.jsonl

[lemyx]

Dear @howard-yen,

What are the mearnings of 1000, k1000, dep3, accordingly?

Each example within within the file has 1000 contexts, is this the meaning fo first 1000?

---

Then as metioned in the paper,

Given an input length L, we first determine the number of passages k that can fit within L tokens, then retrieve k passages[3] from the corpus[4] that do not contain the answer as distractors

If the k1000 corresponds to the number of needed passges to fit the L tokens?

---

What is the mearning of dep3?

What is the function of hard negative passages here?

---

Best wishes

[howard-yen]

Hello, thanks for the question. For the case of HotpotQA, k denotes the number of retrieved passages, so k1000 means that each query contains 1000 passages in the input context. These 1000 passages are just the top retrieved passages (after filtering out the passages that contain the gold answer) and correlate with the specified input length.

dep3 means the number of different depths each query has. For reference, datasets like NQ and TriviaQA have exactly one gold passage for each query, and we would put that gold passage at six evenly spaced depths (if you notice those files have dep6). However, since HotpotQA has multiple gold passages and it's unclear how we should space them out, we simply shuffle all passages into three random permutations to simulate gold passages at different depths (hence dep3).

As for the function of hard negative passages—it's the same as the other RAG settings; the hard negatives serve as distractors from the gold passages. This makes the tasks harder than simply using random contexts.

Let me know if you have any further questions!

[lemyx]

Dear @howard-yen , thanks for your timely and patient reply! After reading your response and executing the codes for generating the HotpotQA dataset, here are some additional questions waiting for your guidance hopefully. Thank you once again!

---

Question 1: How does the hard_negative_ctxs function?

import json
path = "data/kilt/hotpotqa-dev-multikilt_1000_k50_dep3.jsonl"
data = []
with open(path, "r") as f:
    for line in f:
        data.append(json.loads(line))

for _data in data:
    for _hard_negative_ctxs in _data['hard_negative_ctxs']:
        assert _hard_negative_ctxs in _data['ctxs'] # AssertionError

assert len(data) == 2961

During the prompt construction process, the ctxs are converted into passges. Since the hard_negative_ctxs is not guaranteed to be included in the complete ctxs, the initial thought considering hard_negative_ctxs may not function as distractors from the gold passages.

---

Question 2: Has dep3 reflected into the synthesized dataset yet?

questions = set([_data['question'] for _data in data])
assert len(questions) == len(data) // 3

Since each unique question corresponds to exactly three samples within the test file, so can we injecture that the dep3, i.e., the three random permutations has been reflected in the synthesized dataset yet?

---

Question3: What is the meaning of the first 1000 in the test filename of data/kilt/hotpotqa-dev-multikilt_1000_k50_dep3.jsonl?

---

Best wishes

[howard-yen]

Q1: The hard negatives are used to construct the passages—these are the passages that were retrieved but do not contain the gold answer (i.e., by substring exact match). The resulting ctxs contains a subset of the passages in the hard negatives + the gold passages so it's not expected for every hard negative to be in ctxs.

Q2: Yes, if you look at the data preprocessing code

HELMET/data.py

Line 92 in 87bbd6d

def update(sample):

You'll find that we keep all samples with the same question, which should have dep number of samples total. You should be able to use the given data files without any additional work.

Q3: the first 1000 simply means that there is a total number of 1000 contexts in ctxs