Topic modelling

[lukavdplas]

It would be possible to train a topic model for a corpus (offline) and offer visualisations etc.

Training the topic model is not really an issue, but we should have an idea of what we want to do with the trained model.

Some things to consider:

• The topics that come out of a model are very abstract - they are essentially probability distributions over the vocabulary. It is not trivial (or uncontroversial) to link topics to clear labels (e.g. 'politics'), so you are morely to end up with descriptions like 'cluster 3: minister, president, MP, ...'. It is always good to be cautious with linking topics to real mental concepts.
• A fair amount of research into topic modelling is more interested in the model itself. How many topics are optimal, which terms are considered the most 'topical', etc. But this is not very interesting if you were not involved with the training process. It is also not interesting if you do not have experience with topic modelling.
• Our target audience includes a lot of beginner users who are not familliar with topic modelling. What results can we present that are still valuable to them?

Some possible applications:

• Given a document, show documents with a similar topic distribution.
• Show the density of one or all topics in a bar chart, compared to the date or some other field. For instance, cluster 3 seems to be about politics, then show a bar chart of how much this topic features with different newspapers.
• Plot (a random sample of) documents in a scatter chart using PCA. Then let the user pick a field to color-code the documents. For instance, colour based on the genre, newspaper, or date. This can tell you whether a particular field is strongly correlated with topics.

[BeritJanssen]

I wonder whether this is not actually one of the cases where we'd better advise users to use dedicated software, so that they can finetune settings, such as number of topics.