Yikai-Liao / efficient_bpe Public

Notifications You must be signed in to change notification settings
Fork 2
Star 6

An Efficent BPE Algorithm Faster then Hugging Face Tokenizer's Implementation

6 stars 2 forks Branches Tags Activity

Name		Name	Last commit message	Last commit date
Latest commit History 11 Commits
data		data
output		output
tmp		tmp
README.md		README.md
bpe_eval.ipynb		bpe_eval.ipynb
ebpe.py		ebpe.py
ebpe_v2.py		ebpe_v2.py
requirements.txt		requirements.txt

Repository files navigation

Efficient BPE

博客高效中文BPE实现代码仓库。

ebpe.py 我的高效BPE实现
bpe_eval.ipynb 评测代码
requirements.txt 为 bpe_eval.ipynb的依赖，ebpe.py 本身仅以来 tqdm 显示进度条