Topic Modelling for Short Text

Over the past few years, our increased ability to store large amounts of data, coupled with the \nincreasing accessibility of the internet, has created massive stores of digital information. Consequently, \nit has become increasingly challenging to find and extract relevant information, thus \ncreating a need for tools that can effectively extract and summarize the information. One such \ntool, is topic modelling, which is a method of extracting hidden themes or topics in a large collection \nof documents. \nInformation is stored in many forms, but of particular interest is the information stored as short \ntext, which typically arises as posts on websites like Facebook and Twitter where people freely \nshare their ideas, interests and opinions. With such a wealth in data and so many diverse users, \nsuch stores of short text could potentially provide useful information about public opinion and \ncurrent trends, for instance. Unlike long text, like news and journal articles, one of the commonly \nknown challenges of applying topic models on short text is the fact that it contains few words, \nwhich means that it may not contain sufficiently many meaningful words. \nThe Latent Dirichlet Allocation (LDA) model is one of the most popular topic models and \nit makes the generative assumption that a document belongs to many topics. Conversely, the \nMultinomial Mixture (MM) model, another topic model, assumes a document can belong to at \nmost one topic, which we believe is an intuitively sensible assumption for short text. Based on this \nkey difference, we posit that the MM model should perform better than the LDA. \nTo validate this hypothesis we compare the performance of the LDA and MM on two long text \nand two short text corpora, using coherence as our main performance measure. Our experiments \nreveal that the LDA model performs slightly better than the MM model on long text, whereas the \nMM performs better than the LDA model on short text.