Handling duplicates in the datasets

[rfbr]

Hello team!

First of all, thank you for your excellent work.
I have a question regarding the potential duplicates in the datasets that could impact the benchmark's reliability.
While examining the load_qa method, you wrote the following:

if max_test_samples is not None:
    # some datasets do not have id (e.g., nq), so we assume unique questions
    keys = set(data[key])
    keys = random.sample(sorted(keys), min(max_test_samples, len(keys)))
    data = data.filter(lambda x: x[key] in keys)

The comment suggests that questions are assumed to be unique. However, when checking the actual data, this assumption doesn't appear to hold. For example, analyzing nq-dev-multikilt_1000_k50_dep6.jsonl shows that while the file contains 5958 lines (wc -l nq-dev-multikilt_1000_k50_dep6.jsonl), there are only 993 unique questions (jq -r '.question' nq-dev-multikilt_1000_k50_dep6.jsonl | sort -u | wc -l).
Shouldn't the loading logic be updated to deduplicate questions before sampling? Because right now, the same question might be sampled multiple times which might skewed the evaluation.
I'd appreciate some clarification, thanks!