Paraphrase detection is the task to define whether two formally different texts have the same (or similar) meaning. This notebook deals with sentence-level paraphrases from Paraphraser corpus.
The Paraphraser corpus contains sentence pairs obtained from the Russian news headlines (РИА Новости, Лента.ру, Российская газета, РБК) with crowdsourced annotation Pivovarova et al., 2017.
Paraphrase classes include:
- 1 - non-paraphrases
- 0 - loose paraphrases (similar meaning)
- 1 - strict paraphrases (same meaning)
No specific criteria on the distinction between loose and strict paraphrases are provided by the creators of the corpus, so we take it as is.
In the notebook with paraphrase detection algorithm we try to detect 7 syntactic and 7 lexic transformations which lead to a paraphrase. Some functions which detect a transformation require txt files that are stored in data (they are also incorporated into the ipynb file so you don't have to dowload them to run the code). Each sentence pair is viewed as a number of transformations. These transformations serve as features for the future classification model.
We perform two-class and three-class classification with a number of models.