Add neural network model

[andrewtavis]

This issue is for adding an embeddings neural network implementation to wikirec. This package was originally based on the linked blog post, but the original model implementation to now has not been included. That original work and the provided codes could serve as the basis to adding such a model to wikirec, which ideally would also be included in the documentation and tested. That model was based on analyzing the links between pages, which could serve as a basis for the wikirec version with modifications to wikirec.data_utils, or the model could focus on the article texts. Partial implementations are more than welcome though :)

Please first indicate your interest in working on this, as it is a feature implementation :)

Thanks for your interest in contributing!

[victle]

I'm personally trying to learn more about neural networks, so I'd love to work and contribute to this in pieces. I took a quick skim through the linked resources, and there's use of t-SNE as well to visualize the books. I know that you're looking to put together t-SNE for wikirec as well #35, so in the future that could definitely be something I can work on as well. Let me know what you think is a good first step!

[andrewtavis]

For the neural network model the question would be if we implement the method from the blogpost directly where it's looking into the links to other Wikipedia pages, which would then require the cleaning process to have an option to prepare it in this way. Websites URLs are being removed as of now, which maybe they shouldn't? We could of course devise another method though :)

We already have pretrained NNs covered with BERT, so a NN approach that tries to create embeddings might be wasted effort as that's a lot of computing to try to beat something that's explicitly trained on Wikipedia in the first place.

Another model that's popped up in the last few years is XLNet, which I guess would be the other natural implementation to look into.

Let me know what you think on this :)

[victle]

I think dealing with the links themselves might be a good first approach instead of training a whole new NN. I'll look into that blogpost and see if I can find how links are explicitly addressed, unless you had any thoughts on that?

I'll have to look into XLNet as it does look interesting! Though I am definitely lacking in terms of raw computing power.

[andrewtavis]

The original blogpost author finds the wikilinks in the data preparation steps, which are shown in this notebook. He's using wiki.filter_wikilinks() from mwparserfromhell, which we also have as a dependency. Basically in his cleaning steps he's getting titles, the infobox information, the internal wikilinks, the external links, and a few other diagnostic features, whereas we're just getting titles and the texts themselves.

Implementing this his way would honestly be a huge change to the way the cleaning works, so maybe the best way to go about this is give an option in wikirec.data_utils.clean where the websites would not be removed (might be best anyway), and we could use string methods or regular expressions to extract the links from the texts themselves. For this it'd basically be finding instances of https://en.wikipedia.org/wiki/Artcile_Name by matching the first part, and then we'd need to just extract Article Name and make sure there's a way to not have repeat entries.

Once we have those it's basically following the original blogpost :)

[andrewtavis]

For XLNet it looks like we'd be able to use sentence-transformers like the BERT implementation, so the big question then becomes what a suitable model would be. It'd be loading in a huggingface model, as sentence-transformers allows those models to be loaded along with its BERT models. I'm not sure if it's similar to BERT where there are multilingual models and ones that are better for certain use cases or others.

References for this are the XLNet documentation from huggingface and this issue so far.

[victle]

@andrewtavis Hey! Just wanted to update, I've been pretty busy, but am still wanting to work on this issue. I wanted some clarification on what we can do to incorporate these different cleaning methods. As it is now, we're just grabbing the text of the article, which includes the text displayed for the internal wikilinks (I don't think they're getting cleaned out, are they?) How would grabbing the wikilinks themselves substantially improve the "performance" of the recommendation models, since we already have the text from the links' names as part of the inputs to the models already?

[andrewtavis]

@victle, hey :) No worries on a bit of silence :)

Thing is that the URLs are being cleaned as of now. As seen at this line, the websites are being removed, but not the texts that they're the links for. I'm thinking now that this is a random step that actually doesn't need to be included in the cleaning process. We could simply remove this, and then you could then extract the internal Wikipedia links from the parsed text corpuses.

Grabbing the links themselves would basically just make a new modeling approach. The assumption would shift from "I believe recommendations can be made based on which articles have similar texts" to "... which articles are linked to similar things." The second assumption is the one from the blog post, and he also got strong results, so we could implement that approach here as well :)

It kind of adds another layer of depth to a combined model as well. Right now we can combine BERT and TFIDF and get something that accounts for semantic similarity (BERT) and explicit word inclusions (TFIDF), both of which as of now are working well and even better when combined. This could give us a third strong performing model that could add a degree of direct relatedness to other articles. To combine it with others the embeddings would need to be changed a bit though, as his approach embeds all target articles and all that they're linked to. Could be a situation where we could simply remove rows and columns of the embeddings matrix based on indexing though.

I checked our results against the ones he has in the blogpost. The direct way that this could help is books that are historical in context. So far we've been picking fantasy novels for examples, which ultimately seem to be performing well as they would have unique words and places that lead to books by the same author. An example is his results for War and Peace:

Books closest to War and Peace:

Book: Anna Karenina               Similarity: 0.92
Book: The Master and Margarita    Similarity: 0.92
Book: Demons (Dostoevsky novel)   Similarity: 0.91
Book: The Idiot                   Similarity: 0.9
Book: Crime and Punishment        Similarity: 0.9

Our results for a combined TFIDF-BERT approach are:

[['Prince Serebrenni', 0.6551468483971056],
 ['Resurrection', 0.6545365970449271],
 ['History of the Russian State from Gostomysl to Timashev',
  0.6189035165863549],
 ['A Confession', 0.6068009160763292],
 ['August 1914', 0.5945890587900238],
 ['The Tale of Savva Grudtsyn', 0.5914113595267484],
 ['The Don Flows Home to the Sea', 0.5905929292423463],
 ['Anna Karenina', 0.5887544831477559],
 ['Special Assignments', 0.5798599047827274],
 ['Russia, Bolshevism, and the Versailles Peace', 0.5791726426273041]]

They're all classic Russian books, but his results are "better" in my opinion. We get similar results for The Fellowship of the Ring, as expected :)

Sorry for the wall of text 😱 Let me know what your thoughts are on the above!

[victle]

I want to try and summarize my understanding below:

After reading that first paragraph, this is my understanding. I'm using the beginning text of Prince Serebrenni as an example 😄 Before cleaning the raw text, you'll get something like "Prince Serebrenni (Russian: Князь Серебряный) is a historical novel by [[https://en.wikipedia.org/wiki/Aleksey_Konstantinovich_Tolstoy]](Aleksey Konstantinovich Tolstoy)...". And through that line you referenced to in wikirec.data_utils.clean, it's simply removing the link entirely, such that only "Aleksey Konstantinovich Tolstoy" is left?

So then, if we implement this third approach/model that looks at the internal links, we could combine that with TFIDF and BERT to make an overall stronger model (hopefully!) Although, I'm not knowledgeable to know how you would combine embeddings, so that might need more explanation 😅

Either way, it does sound interesting and challenging! I think I'll begin with removing that cleaning step, and replicating the approach from the blog post to extract the links. Does that sound like a reasonable starting step?