replace tokenizer with more compatible js-tiktoken

transitive-bullshit released this 31 May 02:42

· 1 commit to main since this release

Merge pull request #571 from masterkain/js-tiktoken 729b5a4
switch tokenizer implementation with pure js and more compatible js-tiktoken 3ec62b8

Thanks @masterkain and @dqbd 🙏

v5.2.4...v5.2.5

Contributors

masterkain and dqbd

Assets 2