Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Managing model specializations/variants (flavors) #1151

Draft
wants to merge 18 commits into
base: master
Choose a base branch
from
Draft

Conversation

kermitt2
Copy link
Owner

@kermitt2 kermitt2 commented Aug 4, 2024

This PR introduces a simple management of alternative models to use when processing a document. A model variant is for example a alternative header model trained with its own training data and labels (to cover documents with specific header section different from scholar articles), or an alternative segmentation model for segmenting something else than scholar papers.

To process a document with alternative model(s), we use a string called "flavor" to identify it. If the flavor is indicated, the selected model will use the "flavor" model if it exists, and the normal model if the flavor does exist for this model (so defaulting back then to the standard models).

Flavor model training data are always located as sub-directory of the standard training data path, e.g. for the flavor "sdo/ietf", the training data of the header model for this flavor will be under grobid-trainer/resources/dataset/header/sdo/ietf. The training data of the segmentation model for this flavor will be under grobid-trainer/resources/dataset/segmentation/sdo/ietf, and so on.

For running grobid following a particular flavor, we add the flavor name as additional parameter of the service:

curl -v --form input=@./nihms834197.pdf --form "flavor=sdo/ietf" localhost:8070/api/processFulltextDocument

Grobid will then solve the right models to use given the Grobid model hierarchy/cascade: the flavor ones if they exist, or the default ones if not.

In this branch, "sdo/ietf" flavor is just used as an example, the corresponding flavor models are not actually really working.

@kermitt2 kermitt2 marked this pull request as draft August 4, 2024 09:44
@lfoppiano lfoppiano added this to the 0.8.2 milestone Aug 4, 2024
@lfoppiano
Copy link
Collaborator

Thanks!

I'm going to test it directly in the branch for the light-segmentation models :-) and feedback changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants