Running CWQ

[KyuhwanYeom]

Hi,

I tried to run inference on CWQ, but faced some challenging issues.

For step 1: Graph-constrained decoding, there is hyper parameter index_path_length as default value of 2.
Especially, when I run scripts/graph_constrained_decoding.sh as given default setting, its performance on CWQ is fairly low. (about Hits@1 62, F1-score 52)

When I changed this value to 4 and run decoding, too much time is to be consumed (about 2 weeks).
Should I change this value to 4 for CWQ? Or is are any other solution?

Best regards,
Kyuhwan

[RManLuo]

Hi, Thanks for your interest in our work. I think your current settings are correct. Have you tried to run graph_inductive_reasoning.sh to see the result? This is used to report the final results in the paper.

[KyuhwanYeom]

Thanks for your reply.

We ran graph_inductive_reasoning.sh followed by graph_constrained_decoding.sh.
For inference, we did not change any setting in both of script files.

step1_result3090_GCR.txt
step2_result_GCR_gpt.txt

We provide our output log files for convienence.

Best regards,
Kyuhwan