Commit 9209c29

authored

more domain specific tokens (#585)

1 parent 1f86da0 commit 9209c29Copy full SHA for 9209c29

5 files changed

+234

-2

lines changed

apps/api.nameai.dev
- nameai/data
  - domain_specific_dictionary.txt
  - tests
    - tokenization_quality.json
- pyproject.toml
- tests
  - test_ngrams.py
  - test_tokenizer.py

5 files changed

+234

-2

lines changed

Comments

(0)