Automated Hashtag Hierarchy Generation Using Community Detection and the Shannon Diversity Index, with Applications to Twitter and Parler

Developing semantic hierarchies from user-created hashtags in social media can provide useful organizational structure to large volumes of data. However, construction of these hierarchies is difficult using established ontologies (e.g. WordNet [C. Fellbaum (ed.), WordNet: An Electronic Lexical Database (MIT Press, Cambridge, MA, 1998)]) due to the differences in the semantic and pragmatic use of words versus hashtags in social media. While alternative construction methods based on hashtag frequency are relatively straightforward, these methods can be susceptible to the dynamic nature of social media, such as hashtags with brief surges in popularity. We drew inspiration from the ecologically based Shannon Diversity Index (SDI) [J. L. Wilhm, Use of biomass units in Shannon’s formula, Ecology 49(1) (1968) 153–156] to create a more representative and resilient method of semantic hierarchy construction that relies upon network-based community detection and a novel, entropy-based ensemble diversity index (EDI) score. The EDI quantifies the contextual diversity of each hashtag, resulting in thousands of semantically related groups of hashtags organized along a general-to-specific spectrum. Through an application of EDI to social media data (Twitter and Parler) and a comparison of our results to prior approaches, we demonstrate our method’s ability to create semantically consistent hierarchies that can be flexibly applied and adapted to a range of use cases.