Avaliação da sensibilidade de métricas de avaliação de tópicos

Abstract

The growing tendency of store all the knowledge and content produced digitally makes it increasingly difficult to find all this information and organize it. Topical modeling algorithms allows to extract topics from vast collections of documents. A document, which maybe short or long, can be defined as a mixture of topics, which are a set of words sorted by their probability ofoccurrence. Evaluating the quality of a topic is atask that can be simple for humans,al though it is very expensive when dealing with large amounts of data. Thus, computational methods, known as coherence metrics, are used to measure the quality of topics from the co-occurrence between the words that compose them. However, different metrics can generate different results when applied to the same topic. This work will evaluate the sensitivity of some of these metrics by applying them to a set of topics that have been created, adulterated by inserting intrusive words, and validated by humans. As result, in general, the metric CUICI was shown to be the most sensitive, while the metrics CV and CUMASS were shown to be the least sensitive.