[feat] Integrating W&B Tables for Prediction Visualization

[ayulockin]

🚀 Hey all, this PR adds a little feature on top of the WandbLogger that I have improved/extended in this PR, #1137. The PR adds the prediction report generated when evaluation.predict=true as W&B Tables.

What is W&B Tables?

It's an interactive way of going through your dataset and model prediction. It is a two dimensional grid of data where each column has a single type of data—think of this as a more powerful DataFrame. Tables support primitive and numeric types, as well as nested lists, dictionaries, and rich media types. Log a Table to W&B, then query, compare, and analyze results in the UI.

(Source)

Modification

This PR adds a log_prediction_report to the WandbLogger. This method is called in the trainer/core/evaluation_loop.py file (a 2-3 line addition).

Usage

To use this, in the config/defaults.yaml do, training.wandb.enabled=true, training.wandb.log_tables=true and evaluation.predict=true.

Result

Here's the Tables page that you can check out.

Superpowers

One can now keep track of the model prediction result. I found keeping track of the same using json or csv challenging especially when I am revisiting the project after few days.

What more can be done?

So far all good, but W&B Tables has a lot to offer. We can log the actual input question and image instead of question_id and image_id. Any help in this direction would be useful.

Future additions/improvement

Here's a crude road map for this feature.

The following is how I see it progressing, which might change in the future.

• Log the prediction report as W&B Tables. The current solution is generic in my opinion.
• Add ground truth answer for the given dataset and see if images are available to be logged as a column.
• Add ability to log media.

I would love to know if you are interested in this. :D

[ayulockin]

Hey @ebsmothers, @apsdehal and team I would love to get some feedback on this PR, especially on the crude roadmap:

[emanuelevivoli]

wait for it