Releases: niieani/gpt-tokenizer
Releases · niieani/gpt-tokenizer
2.4.0
2.4.0 (2024-09-23)
Features
- performance optimizations (661e283)
2.3.0
2.3.0 (2024-09-20)
Features
- improve performance, memory usage & initialization time (#50) (e2c560a), closes #18 #35
2.2.3
2.2.3 (2024-09-16)
Bug Fixes
- deps: update dependency rfc4648 to ^1.5.3 (fcbf48a)
2.2.2
2.2.2 (2024-09-15)
Bug Fixes
- improve test typing (bbd0764)
- upgrade dependencies (including typescript) (75ebd54), closes #49
2.2.1
2.2.1 (2024-07-18)
Bug Fixes
- add files for other models (c21d498), closes #19
- regenerate o200k encoding from tiktoken file (c7ba009)
2.2.0
2.2.0 (2024-07-17)
Bug Fixes
- add correct encoding for o200k_base (137e07b)
- add gpt-4o on readme as supported model (27b4e20)
- update readme (0b33e1e)
Features
- add o200k_base test plans (44ce38e)
- added o200k_base to encodings and configured it's specialTokens (2a9da2b)
2.1.2
2.1.2 (2023-10-07)
Bug Fixes
- bind encodeChat and encodeChatGenerator (86c270c), closes #15
2.1.1
2.1.1 (2023-06-01)
Bug Fixes
- missing default exports due to limitations of ESM (2a55474), closes #11
2.1.0
2.1.0 (2023-06-01)
Bug Fixes
- correct excluded path (71af9d3)
- workaround for webpack not exposing the default export in UMD correctly (84887b4), closes #12
Features
2.0.0
2.0.0 (2023-05-24)
Features
- complete rewrite to support different models (eedd944), closes #5 #6
- more tests and better README (e660a25)
BREAKING CHANGES
- default encoder is now GPT3.5 / GPT4