咨询如何构造更长的测试集

[anda4]

想要咨询如何构造更长的测试集， 例如如何构造1M的测试集合

[Leolty]

Hi @anda4,

Thanks for asking!

This is possible, but it might be somewhat hard and biased since most files won't have such a large number of dependencies.

My initial suggestion would be to construct the test set by simply including all GitHub files in the context, without performing any parsing or retrieval. This would result in a significantly long context for many github repos, although much of the included information may be irrelevant.

[anda4]

Thank you. I have mixed some irrelevant files into context to build longer context. The experiments show that the LLM can give the right answer for some testcases, thus I believe this needle-in-the-hay like benchmark is kind of usable.

…

From: "Tianyang ***@***.***> Date: Sat, Sep 21, 2024, 05:51 Subject: Re: [Leolty/repobench] 咨询如何构造更长的测试集 (Issue #19) To: ***@***.***> Cc: ***@***.***>, ***@***.***> Hi @anda4<https://github.com/anda4>, Thanks for asking! This is possible, but it might be somewhat hard and biased since most files won't have such a large number of dependencies. My initial suggestion would be to construct the test set by simply including all GitHub files in the context, without performing any parsing or retrieval. This would result in a significantly long context for many github repos, although much of the included information may be irrelevant. — Reply to this email directly, view it on GitHub<#19 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BJUUFDTB3QRWXJ4MRFQXIGLZXSKEZAVCNFSM6AAAAABOPIUHZKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNRUGY3DINZRGU>. You are receiving this because you were mentioned.[image: https://github.com/notifications/beacon/BJUUFDSGTYMPOLCRFYILDRLZXSKEZA5CNFSM6AAAAABOPIUHZKWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTUM6HVYW.gif]Message ID: ***@***.***>