How to evaluate models?

[liang-hou]

Hi, thanks for the great work. I would like to know how to evaluate the generation performance of models. Specifically, I am interested in how to calculate FID and other metrics such as IS, and whether you have implemented automated evaluation pipelines.

[clementchadebec]

Hi @liang-hou,
Thank you for opening this issue and sorry for the late reply. There is no automated evaluation pipeline for the moment. This is a feature that can be added in the future if you think it would be helpful. To evaluate your models you can use the samplers to generate new images and store them in a dedicated folder. Then, you can use the official implementations (see for instance the repo for the FID) or use torchmetrics which also includes easy metric computation.

I hope this helps,

Best,

Clément

[MiguelCosta94]

Hi @clementchadebec ,
Which network have you used as a feature extractor to calculate the FID (values shown on the paper)?

Best regards,
Miguel

[clementchadebec]

Hi @MiguelCosta94 ,

I used this piece of code to compute the FID values.
https://github.com/bioinf-jku/TTUR/blob/master/fid.py#L271

Best,

Clément