Devising ways to best combine recommendations

[andrewtavis]

This issue is to discuss ways to best combine vector embeddings so that a wikirec user can optimally pass more than one argument to wikirec.model.recommend.

The current way of combining recommendations for more than one input is to simply take the arithmetic means of the similarity matrix rows for each passed title, which is depicted in the following snippet from wikirec.model.recommend:

for i, t in enumerate(titles):
    if t == inpt:
        if first_input:
            sims = sim_matrix[i]

            first_input = False

        else:
            sims = [np.mean([s, sim_matrix[i][j]]) for j, s in enumerate(sims)]

A discussion of whether this is the best way to do this would be much appreciated! Furthermore, how could the above be changed to allow a user to express disinterest (as discussed in #33).

[victle]

Hello! Cool project! I'm sort of new to NLP in general. But, I was thinking for multiple inputs, maybe an approach could be aggregating the sims from different methods (e.g., median, weighted averages), and picking the sims with the highest score?

[andrewtavis]

Hi, and thanks for the praise! Quick aside, sort of new to NLP in general's all good, so no worries :)

Definitely combined recommendations will continue to be some kind of averaging of the scores and then selecting from an ordered list. Your idea of taking the median is an interesting one though - say taking the median value of the similarities for the titles that have already been selected?

With regards to weighted averages, this is actually very interesting to me as I think it'd be great if people could indicate that one item in a list is more important than the others. As of now I'm thinking about how to do this in an intuitive fashion where a user would be able to pass titles of interest and their weights and then get results.

Would be great to get further opinions from you on all this :)

[victle]

Hi, and thanks for the praise! Quick aside, sort of new to NLP in general's all good, so no worries :)

Yep! I'll try my best haha.

Median would be an option, especially if the distribution of different titles may be skewed. Though, I'm not sure if this would actually improve "performance".

Off the top of my head, perhaps people can pass an optional argument or keyword argument with a list of "ratings" for each of titles passed in? At least that way it won't break anything?

Either way, I'd love to know if you have any suggestions for how I can start contributing :). Perhaps a start could be implementing different aggregating methods for this snippet in the wikirec.model.recommend function?

[andrewtavis]

If you figure out the way to effectively get ratings into and out of this thing then you and I can start ourselves an open source organization :D Sarcasm that's actually serious aside, I'd be interested in exploring this more, and is something I've already been working on a bit :)

Simplest thing that could be done is, as you said, add a ratings keyword argument to wikirec.model.recommend, with the relative importance of similarities then being weighted by the proportion of the corresponding score to the sum of the passed scores. I've tried things similar but actually not exactly like this before, and the results are good, but the thing to note is that you're not getting valid scores back out of it. All the sims are on [0, 1), so you can't expect to get out a 9/10 when the the 9s you're passing are constantly multiplied by a number < 1. To me the best thing to then do would be to turn off the numerical part of the output so you just get the list, but then you could also experiment with scaling the numerical part to be on [0, 1] or even [0, 10]. The latter would give too many 9.9/10s to be sensible, but then to me it's worth a try.

If you wanted to implement a method and/or ratings kwarg for wikirec.model.recommend, then the effort would be much appreciated! For models, I'd focus on BERT, TFIDF, and LDA, as Doc2vec is just so time intensive and so meh as of now... You'd be more than welcome to create your own notebook in the examples to test these arguments, as I guess as of now I'd prefer to keep the current examples as they are. Your implementations and results would then be welcome additions to the readme :) (thinking out loud, if the results are getting long we should look to add drop downs for comparative results to the readme as I've done with causeinfer's readme).

And speaking of examples, getting examples/rec_movies ran all the way through would also be helpful if you have access to a server that could handle the file sizes (no biggie if not, as I expect someone will come along who can run these larger examples).

Great to have you!

[victle]

Thanks for the informative answer! An open source org is not out of the question 😆. I'll take a crack at putting together a basic framework that will take ratings in and then manipulating the sims. And then, putting together an examples to test them. I do have to refresh myself on BERT, TFIDF, and LDA (would appreciate any resources on that), but I think the methods themselves should be pretty intuitive based on your examples.

[andrewtavis]

I'll put together some resources for the models and send them along tomorrow :) Maybe trying the Wiki feature of repos would be useful in this regard 🤔

For the Jupyter notebook, in case you don't use them already check out the notebook extensions, which will make the example a lot easier as far as an auto ToC with Table of Contents (2), Jupyter Black (as the code format is Black, but don't stress about this!), Highlight Selected Word, spellchecker (for markdown cells), and Execute Time - with these being the main ones I use :)

Also, if you're using VS Code for your editor and are looking for extension suggestions to similarly make it all easier, I'd be happy to shoot some along!

[andrewtavis]

@victle, the package Wiki now has a page Resources for Models that has what to me were the best descriptions and videos for the models, as well as links to the documentation for the Python implementations that wikirec sources :)

[victle]

Thanks, I'll definitely look them over. I've been trying to reproduce some of the examples in the repository, and have been running across a few issues that I've personally noted down. One main problem I've had is that I haven't been able to get the same output from using wikirec.model.recommend(), though I've only tested LDA and TDIDF so far. For reference, this is what I'm getting in my notebook from using LDA, but TDIDF I'm able to reproduce. Not sure if it's a version issue with some packages or anything.

[andrewtavis]

For LDA you have a random_state argument that's going to be a factor in all of this that will change the results :) See the LDA multicore docs. That argument should be able to be passed as a kwarg for model.gen_embeddings, which will then produce the same results each time. The same is true for BERT and Doc2vec, although they might be referred to as random seed instead. You can see np.random.seed(42) in use in the testing base file for wikirec, although looking at that now I really do need to alter the tests so they're checking the models for explicit return values (will do that this weekend).

Machine learning algorithms all (generally) have an aspect of a pseudorandom number generator that's being used to set the initial state(s) that are then optimized in the modeling process, which is why not setting the random state/seed will lead to slightly different results. The path that the optimization takes will also be different given the state/seed.

Those results for LDA look fine to me :D Most of the other HP books, something that makes sense in The Marvelous Land of Oz, and then some stuff that's just ???????? Fine tuning the LDA model given the multicore parameters would be something for later, as MVP wise I was just thinking get things out and see what works best regardless of optimization.

TFIDF isn't a machine learning algorithm (roughly is just counts and division), and thus will produce the same results :)