Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add BPETokenizer #204

Open
gpengzhi opened this issue Sep 11, 2019 · 0 comments
Open

Add BPETokenizer #204

gpengzhi opened this issue Sep 11, 2019 · 0 comments
Labels
enhancement New feature or request topic: data Issue about data loader modules topic: examples Issue about examples

Comments

@gpengzhi
Copy link
Collaborator

There are some subtle differences between BPE implementation in sentencepiece and BPE implementation in subword-nmt. We could probably delete everthing except multi-bleu.perl in texar-pytorch/bin/utils after this one is implemented. Transformer example could be simplified as well. Related issue #180
@gpengzhi gpengzhi added enhancement New feature or request topic: data Issue about data loader modules topic: examples Issue about examples labels Sep 11, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request topic: data Issue about data loader modules topic: examples Issue about examples
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@swapnull7 @gpengzhi