Proposal: Corpus, Document, Sentence, Token, Language components

[rmarronnier]

As you can see browsing Cadmium shards source code, several entities (for lack of a better word) are declared in different locations and in different ways.

This issue is not just a namespace or redundancy issue but we'd benefit by having fundamental classes or structs describing the tokens, sentences and documents we're dealing with.

I've started in the pos_tagger declaring such structs and objects. It's a WIP and things might change as we'll discover what we need and don't need for higher levels text processing functions.

What's obvious to me is the neat way the Language is declared in cadmium_tokenizer. As languages information is used in different shards (especially the language codes) we could move a big part of it in a language module in cadmium_utils and keep some specific language infos (abbreviations, tag maps, etc) in their respective shards.

More examples from the top of my head :

The Document struct or class might benefit from the cadmium_tfidf and vice versa.

The Cadmium::Utils::Sentence might be renamed to Sentencizer (is that a word ?)
so that a Cadmium::Sentence might exist without conflict.

I'd like to point out that having these classes or structs won't impede users to process raw text without creating these objects. But they are needed to keep morpho-syntaxic infos about the tokens and sentences.