-
Notifications
You must be signed in to change notification settings - Fork 118
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Why SentencePieceTokenizer can't save vocab file #282
Comments
We deleted
|
@gpengzhi |
Could you write down how you integrate |
I want to use vocab file in PairedDataloader, but the the save_vocab function of SentencePieceTokenizer only save the model file.
The model file can't be load by Dataloader because of decoding error.
In sentencepiece_tokenizer.py, I saw you delete the vocab file.
The text was updated successfully, but these errors were encountered: