-
Notifications
You must be signed in to change notification settings - Fork 737
Issues: huggingface/tokenizers
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Author
Label
Projects
Milestones
Assignee
Sort
Issues list
"Solution" to memory hogging in train_new_from_iterator with a hack
#1546
opened Jun 4, 2024 by
morphpiece
How can I get the mapping relationship between byte values and Unicode characters of the fast tokenizer?
#1545
opened Jun 4, 2024 by
LuoKaiGSW
Training HuggingFace tokenizer - ignore_merges
Feature Request
planned
#1537
opened May 22, 2024 by
ykoyfman
How to allow the merging of consecutive newline tokens \n when training a byte-level bpe tokenizer?
#1534
opened May 18, 2024 by
liuslnlp
How to Batch-Encode Paired Input Sentences with Tokenizers: Seeking Clarification
#1531
opened May 14, 2024 by
insookim43
Special token handling breaks idempotency of sentencepiece due to extra spaces
#1527
opened May 9, 2024 by
cat-state
Link to download the training text in
docs/source/quicktour.rst
is broken
#1526
opened May 9, 2024 by
14jdelap
Error: Cannot find module 'tokenizers/bindings/tokenizer'
Stale
#1522
opened May 6, 2024 by
meichangsu1
UnigramTrainer: byte_fallback is false.
Feature Request
training
#1515
opened Apr 25, 2024 by
Moddus
BPE Trainer doesn't respect the
vocab_size
parameter when dataset size is increased
Stale
#1514
opened Apr 25, 2024 by
Abhinay1997
Extended vocab tokenizer merging text into a single string without spaces while decoding
#1501
opened Apr 17, 2024 by
savanth14
Previous Next
ProTip!
Find all open issues with in progress development work with linked:pr.