Custom model API endpoint support

[WiktorSobanski]

Hi, I am wondering if there is possibility to use langtest with custom endpoints - instead of passing hub parameter for harness object, I would like to somehow pass API endpoint URL. Is it supported in current version of langtest?
Thanks!

[chakravarthik27]

@WiktorSobanski,

I understand your request, but unfortunately, in the current version of langtest, it is not possible to directly pass a custom API endpoint URL instead of the "hub" parameter. However, we are actively working on including this feature in the upcoming release. If you are under time pressure, you can upload the model to Hugging Face and start testing with langtest.

thank you

[chakravarthik27]

Hello @WiktorSobanski!

Great to hear you're interested in leveraging langtest with custom endpoints. In the latest release, we've introduced an exciting feature that allows you to pass API endpoint URLs directly, offering more flexibility in your implementation. You can now seamlessly integrate your custom endpoints, enhancing the functionality of langtest to suit your specific requirements. Feel free to explore this new capability and let us know if you need any further assistance. Happy testing!

pip install langtest

harness = Harness(task="text-classification",
                  model={'model': model, "hub": "custom"}, data={'data_source': 'path/to/data'})
 

for more detail: Review this PR

[WiktorSobanski]

Hi @chakravarthik27! That's amazing! Thanks a lot for keeping me notified!