Skip to content

Issues: openai/tiktoken

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
0 Open 101 Closed
0 Open 101 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Cannot tokenize byte sequences that are not valid UTF-8 due to design flaw
#388 by sharpobject was closed Mar 8, 2025
7
build wheels for musl aarch64
#386 by codefromthecrypt was closed Mar 1, 2025
2
Missing git tag
#382 by frenzymadness was closed Feb 14, 2025
1
ValueError: tiktoken plugin tiktoken_ext.openai_public does not define ENCODING_CONSTRUCTORS
#377 by renata-gotler was closed Feb 21, 2025
2
Add o3-mini support
#374 by Nicba1010 was closed Feb 14, 2025
11
How to use in full offline env?
#369 by momomobinx was closed Jan 15, 2025
1
tiktoken.encoding_for_model fails for o1
#367 by Somerandomguy10111 was closed Jan 28, 2025
Tokenizer for o1
#363 by texasroh was closed Feb 14, 2025
4
ValueError: not enough values to unpack (expected 2, got 1).
#353 by Itime-ren was closed Feb 14, 2025
8
Python 3.11 wheel aarch64 missing for tiktoken 0.8
#347 by hauntsaninja was closed Oct 4, 2024
2
Does tiktoken count only input tokens or output tokens as well?
#344 by GildeshAbhay was closed Oct 3, 2024
1
Is there a new tokenizer for o1 models?
#337 by jiadingfang was closed Oct 3, 2024
9
chatgpt-4o-latest is not yet added
#335 by jvlinsta was closed Oct 3, 2024
3
Facing erros in importing the o200k_base
#334 by JaynouOliver was closed Oct 3, 2024
9
Leveraging DP for bpe_encode function
#333 by lordgavy01 was closed Aug 22, 2024
ValueError: not enough values to unpack (expected 2, got 1) when tiktoken.get_encoding("cl100k_base")
#330 by hzh12345678 was closed Oct 6, 2024
2
When I was inputting long text into a large model, that is, when the len of the text was 1024*1024, a StackOverflow error occurred.
#327 by YangQiangli was closed Feb 16, 2025
2
RecursionError: maximum recursion depth exceeded while calling a Python object
#323 by Hudrolax was closed Jul 19, 2024
5
Support for GPT 4o
#305 by jcrupi was closed Jun 10, 2024
1
Unknown encoding gpt2
#301 by aryagxr was closed May 27, 2024
1
o200k_base pretokenizer - regex error?
#298 by AmitMY was closed May 18, 2024
2
gpt-4o tokenizer
#295 by nxfi777 was closed May 13, 2024
1
Combining marks and indic vowel marks within words are being split breaking all indic languages and most languages except English and CJKs
#292 by ajaykg was closed May 5, 2024
4
Custom tokenizer fails to encode despite characters being in mergeable_ranks
#289 by afang-story was closed Oct 3, 2024
3
Using offline: .tiktoken file gets deleted automatically on Linux
#279 by nkilm was closed Aug 22, 2024
4
ProTip! Adding no:label will show everything without a label.