Skip to content

can not read yttm trained model #8

Description

@fkurushin

HI!

I want to read previously trained bpe model using yttm python module, with your lib, this is my code snippet:

bpeModel, err := os.Open("data/yttm.model")
reader := bufio.NewReader(bpeModel)

m, err := bpe.ReadModel(reader)
if err != nil {
print(fmt.Errorf("failed to read model %v", err))
}

config := bpe.NewConfig(false, false, false)
fmt.Println(m.EncodeSentence("мама мыла раму", *config))

and I have Broken input: unexpected EOF. This error appears on this line of code

logrus.Error("Broken input: ", err)

As I understood there may be some incompatibles between yttm model dump file extension and you ReadModel method? Have I train the model with some specific parameters or load it in go with some extra parameters?

P.S. I also need to mention that I encountered an import error while loading the configuration, so I implemented the NewConfig method to resolve it.

Thanks a lot

UPD: as I understood yttm load model as a txt file of char2id and then merges, but your package loads as a binary, so they are incompatible.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions