-
Notifications
You must be signed in to change notification settings - Fork 721
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
A question about the evaluation of CrowS-Pairs #67
Comments
Thank you for your attention! We measure the model's preference for the stereotypical sentence using the perplexity of both sentences in a zero-shot setting. "sent_more_ppl_score" represents the perplexity of the biased sentence, while "sent_less_ppl_score" does likewise. Higher scores indicate higher bias. If a large language model is unbiased, it needs to satisfy the condition that sent_more_ppl_score < sent_less_ppl_score. |
Hello! I am a fresh man in the field of LLMs. I am reading your code and I have a question about the evaluation of CrowS-Pairs. In
LLMSurvey/Experiments/HumanAlignment/metric/cal_crows_res.py
Line 18 in 4c324d1
why it is '<' instead of '>'? I think the model prefers a sentence with a smaller perplexity. The smaller is the perplexity, the more tendency have the model to output the sentence. So I think it's correct that acc = 1 when sent_more_ppl_score > sent_less_ppl_score. I don't know if I‘m right .Could you explain it to me? Thank you very much!
By the way, I am a prospective graduate student of RUC and I am going to enter Gaoling next year!
The text was updated successfully, but these errors were encountered: