Skip to content

Issues: openai/tiktoken

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
36 Open 218 Closed
36 Open 218 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

FAQ (please read before opening a new issue)
#98 opened Apr 5, 2023 by hauntsaninja updated Apr 6, 2023
Fix missing line numbers in stack traces
#103 opened Apr 13, 2023 by christopherhesse updated Apr 13, 2023
Tiktoken not published to cargo
#24 opened Jan 24, 2023 by zurawiki updated Jul 18, 2023
1
11
Add im_start / im_end in cl100k_base
#48 opened Mar 5, 2023 by spolu updated Aug 12, 2023
4
NPM package
#22 opened Jan 22, 2023 by atgctg updated Sep 25, 2023
7
Performance ideas
#33 opened Feb 20, 2023 by christopher-hesse updated Nov 23, 2023
6
tiktoken error on AWS lambda
#198 opened Sep 21, 2023 by vicky141998 updated Feb 15, 2024
5
tiktoken_ext.__pycache__ Issue On Amazone Lambda
#169 opened Jul 22, 2023 by FarzamHejaziK updated Feb 20, 2024
1
pinging tiktoken URL always fails, pinging api.openai.com always works
#269 opened Mar 14, 2024 by distributev updated Mar 14, 2024
1
Please commit your Cargo.lock file in the repository
#179 opened Aug 17, 2023 by charles-dyfis-net updated Apr 10, 2024
3
Any introduction about api encode_with_unstable?
#137 opened May 26, 2023 by fseasy updated Apr 21, 2024
2
Understanding the intended behaviour of _encode_bytes
#288 opened Apr 29, 2024 by ashleyholman updated Apr 29, 2024
Use a custom exception ValueError subclass for the special tokens warning
#290 opened May 2, 2024 by simonw updated May 2, 2024
Tiktoken educational BPE trainer takes long time to train with vocab size 30k
#299 opened May 19, 2024 by sagorbrur updated May 21, 2024
2
TikToken Tokenizer from scratch ?
#303 opened May 25, 2024 by IsNoobgrammer updated May 25, 2024
How to find the token count of a prompt using meta/llama2-70b model
#274 opened Mar 29, 2024 by pradeepdev-1995 updated Jun 25, 2024
1
Unable to route GET requests through proxy
#259 opened Feb 15, 2024 by aevilevitch updated Jul 29, 2024
2
[FR] Add --offline
#317 opened Jul 6, 2024 by NightMachinery updated Aug 23, 2024
3
SSLError: HTTPSConnectionPool(host='openaipublic.blob.core.windows.net', port=443): Max retries exceeded with url
#281 opened Apr 10, 2024 by sijiashen updated Aug 28, 2024
8
Unofficial bindings / ports in other languages
#97 opened Apr 5, 2023 by hauntsaninja updated Aug 31, 2024
11
F-REQ: If the pip installer doesn't find Rust, it should install the pure python version of the tokenizer
#227 opened Dec 10, 2023 by Emasoft updated Nov 15, 2024
2
Community Resource: AutoTikTokenizer - A Bridge Between TikToken and HuggingFace Tokenizers
#358 opened Nov 7, 2024 by bhavnicksm updated Dec 15, 2024
2
Enhancement: Add convenience token-counting functions to this package
#250 opened Feb 7, 2024 by pamelafox updated Jan 1, 2025
5
tiktoken under counting tokens for openai text-embedding-3-large
#366 opened Jan 9, 2025 by mattkauffman23 updated Jan 17, 2025
2
Optimize _byte_pair_merge function in BPE implementation
#284 opened Apr 12, 2024 by naveens01 updated Jan 20, 2025
1
ProTip! Type g p on any issue or pull request to go back to the pull request listing page.