I'm having trouble running the distill_llama.py.

[foreverboy1989]

I think there's one experiment missing in the experiment config folder: eval_Scrolls.yaml, which calls OurTrainer in the finetune_seq2seq.py in the eval phase, which uses dataset scrolls, but doesn't precede the operation of importing datasets.
Traceback (most recent call last): File "/data1/lxw/code/lolcats/distill_llama.py", line 469, in <module> main() File "/data1/lxw/code/lolcats/distill_llama.py", line 393, in main final_metrics = finetune_trainer.evaluate(model, step=-1, max_batches=None, prefix='final') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data1/lxw/code/lolcats/src/trainer/finetune_seq2seq.py", line 68, in evaluate return self.eval_step(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data1/lxw/code/lolcats/src/trainer/finetune_seq2seq.py", line 94, in eval_step raise ValueError("scrolls_metric is not set in the dataset") ValueError: scrolls_metric is not set in the dataset

[mzio]

hi there! sorry for the late response.

just to double-check, are you getting this error from calling the sample script like ?

python distill_llama.py --model_config distill_llama3_8b_lk_smd_wtk64_fd64_w01 \
--distill_config distill_alpaca_clean_xent0_mse1000_lr1e-2 \
--finetune_config finetune_lora_qkvo_alpaca_clean \
--eval_config eval_alpaca_clean \
...

and does the error only happen at the end of calling the script?

I'm a bit confused by the error message, but what I think is happening is that the metric is not being loaded here

to debug, are you able to:

1. import load_metric from huggingface datasets? i.e.,

from datasets import load_metric 

2. Run this function? in dataloaders/utils/setup.py