Reported Scores in Paper may be low

[lukemcdermotttt]

Table 6 shows Lolcats distilled from Llama 3.1 8B gets 69.7% acc on Winogrande (w/ Llama 3.1 8B gets 73.5% acc)

Then Table 3/4 show that Lolcats distilled from Llama 3 8b gets 74.1% on Winogrande (over Llama 3 8B 73.1% acc), much better.

I reran the LM eval on huggingface checkpoints of Lolcats Llama 3.1 8b (on a tweaked repo, so mine might be wrong) and got winogrande scores that relate closer to your Llama 3 8b-version model. Maybe Table 6 scores should be re-ran to boost performance in paper?

I also might be missing something, the score just seemed odd to me when reading through the paper.