Skip to content

Commit

Permalink
Update README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
aneubeck committed Jul 29, 2024
1 parent 25a90e7 commit b941d43
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion crates/bpe/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -191,6 +191,6 @@ We compared our implementations with the tiktoken implementation on a MacBook Pr
| Heap | 1900 µs ||

As can be seen, our Backtracking implementation beats the TikToken Rust implementation by ~4x.
And even the fully deynamic programming solution is faster with a more consistent runtime.
And even the fully dynamic programming solution is faster with a more consistent runtime.
The tuned heap implementation is still quite competitive to TikToken (especially for smaller inputs).
If the requirement of correct BPE output can be relaxed, then the Greedy approach or the minimal encoding approach are the clear winners.

0 comments on commit b941d43

Please sign in to comment.