Update Evaluation Script for Reranking

[milistu]

Update Evaluation Script for Reranking

Updates

1. New Feature: Added a return_metric argument that specifies which metric will __call__ method return. MRR is still left as default but now we can save the model while training based on NDCG.

2. Documentation Improvement: Updated docstrings for previously missing or incomplete arguments.

Best Regards,

• Milutin

[tomaarsen]

Hello!

I'm actually currently updating the CrossEncoder training flow, and one of the proposed changes that I will introduce is to make the CERerankingEvaluator return a dictionary instead. This will allow you to access both values. Because of this, I'd like to avoid return_metric.

However, if you'd to revert the return_metric part and keep only the docstring changes, then I'd be happy to accept this PR.

• Tom Aarsen

[milistu]

Hi @tomaarsen,

Thank you for your quick response!

I’ve removed the return_metric changes as requested and kept only the docstring updates. Let me know if there’s anything else needed for this PR.

Regarding my other work, I’ve been preparing a larger PR for the CrossEncoder that includes:

• Implementation of a listwise loss function based on the Learning to Rank paper.
• Changes to the .fit method and a new collator to support this.
• An example script for training using the MS_MARCO dataset.

However, I understand you’re currently updating the CrossEncoder training flow, which might conflict with or supersede my changes. I’d love to sync with you to align my updates with your improvements in order to make the process more efficient and avoid redundant work. Let me know how you’d like to proceed—I’m happy to adapt my changes or collaborate further.

Thanks again, and looking forward to your thoughts!

Best regards,
Milutin