Code

Updates to SentencePieceTokenizer#11

Code

Open

msis wants to merge 4 commits intoARBML:masterARBML/tkseem:masterfrom TarteelAI:masterTarteelAI/tkseem:masterCopy head branch name to clipboard

+47-961

Commits

Commits on Jan 4, 2024

• No need to save model to disk while training.

  msis

  committed
  85302f2

  Copy full SHA for 85302f2

• Refactor SentencePieceTokenizer train method to accept additional arguments

  msis

  committed
  8aac9d5

  Copy full SHA for 8aac9d5

• Remove unused import statements

  msis

  committed
  0d6b41e

  Copy full SHA for 0d6b41e

• Refactor SentencePieceTokenizer train method

  msis

  committed
  6c7163e

  Copy full SHA for 6c7163e