Understanding file name and content

[framolfese]

Hi, I have a few questions ( somewhat related to issue #27 ):

1. In the paper, you say that the contexts for each benchmark subset are divided in 8k, 16k, 32k, 64k and 128k. How can I understand the context length from the file names like hotpotqa-dev-multikilt_1000_k20_dep3?

2. In the paper you say that you provide 2 in-context examples for each input question. I was manually inspecting the files like hotpotqa-dev-multikilt_1000_k20_dep3 and I discovered that each question, together with its gold passages and hard negative ctxs, are repeated three times.
   2.1 Which among the three equal samples counts for the performance assessment?
   2.2 Do the other two count as in-context examples? I expect this to be false since it will make the model see the correct answer for the same question, therefore I'd like to understand how you gather in-context examples and what is the purpose of having three equal instances for the same question.
   2.3 Among the fields, there is one called simply "ctxs". I expect this to be the context used for evaluation, whereas "positive_ctxs" and "hard_negative_ctxs" are the ones from which you draw data to create "ctxs". If this is correct, why the objects inside "ctxs" have different shapes and keys? See this one here for example, which has two equal passages but with fields named differently:

"ctxs":[
      {
         "psg_id":12846257,
         "title":"Steampunk",
         "text":"Steampunk\nSteampunk is a subgenre of science fiction or science fantasy that incorporates technology and aesthetic designs inspired by 19th-century industrial steam-powered machinery. Although its literary origins are sometimes associated with the cyberpunk genre, steampunk works are often set in an alternative history of the 19th century British Victorian era or the American \"Wild West\", in a future during which steam power has maintained mainstream usage, or in a fantasy world that similarly employs steam power. However, steampunk and neo-Victorian are different in that the"
      },
      {
         "psg_id":27779800,
         "title":"Lost Horizons (Abney Park album)",
         "text":"Lost Horizons (Abney Park album)\nLost Horizons is the 8th studio album by Abney Park, subtitled \"The Continuing Adventures of Abney Park\". It is the band's first Steampunk themed album.\nTrack listing.\n1. \"Airship Pirates\"\n2. \"The Emperor\u2019s Wives\"\n3. \"Sleep Isabella\"\n4. \"She\"\n5. \"The Secret Life Of Doctor Calgori\"\n6. \"This Dark And Twisty Road\"\n7. \"Herr Drosselmeyer's"
      },
      {
         "id":"27779800",
         "title":"Lost Horizons (Abney Park album)",
         "text":"Lost Horizons (Abney Park album)\nLost Horizons is the 8th studio album by Abney Park, subtitled \"The Continuing Adventures of Abney Park\". It is the band's first Steampunk themed album.\nTrack listing.\n1. \"Airship Pirates\"\n2. \"The Emperor\u2019s Wives\"\n3. \"Sleep Isabella\"\n4. \"She\"\n5. \"The Secret Life Of Doctor Calgori\"\n6. \"This Dark And Twisty Road\"\n7. \"Herr Drosselmeyer's",
         "score":0.81396484375,
         "has_answer":false
      }
   ]

Thanks in advance for the clarifications, as I need to perform inference on your benchmark in a custom way and I need to know exactly how to handle and organize the data in order to have a fair comparison with you.

[howard-yen]

Hi @framolfese

I answered (1) in the issue #33 let me know if that answers your question.

2.1 Which among the three equal samples counts for the performance assessment?

All three samples are evaluated and averaged for the performance.

2.2 Do the other two count as in-context examples? I expect this to be false since it will make the model see the correct answer for the same question, therefore I'd like to understand how you gather in-context examples and what is the purpose of having three equal instances for the same question.

Exactly, they are not used for in-context examples but rather a question from the training set is sampled for the ICL example. You can see the code here

2.3 Among the fields, there is one called simply "ctxs". I expect this to be the context used for evaluation, whereas "positive_ctxs" and "hard_negative_ctxs" are the ones from which you draw data to create "ctxs". If this is correct, why the objects inside "ctxs" have different shapes and keys? See this one here for example, which has two equal passages but with fields named differently:

The reason that they have different keys is that for the retrieved passages, we have the annotation for the score (returned by the retriever), and if it contains the final answer (by exact match). The positive passages that we obtained from kilt's annotations do not have these fields. In the example you showed, it seems like one of the gold passages was retrieved, but doesn't actually contain the final answer. This probably happened in the hotpotqa setting because there are multiple gold passages, and only one of them contains the answer by exact match. In practice, I should have filtered out all gold passages from the retrieved set but hopefully this doesn't affect the results.

Hope this helps, let me know if you have more questions!