Topic Modeling for Customer Service Chats

Chatbot, as a virtual assistant employed by companies, can provide benefits for its users. Users can communicate directly to the chatbot via a short message, after which the chatbot system spontaneously identifies the intent of the message and responds with relevant actions. Unfortunately, the scope of the chatbot knowledge is limited in handling messages by an increasingly varied group of users. The main consequence of such variation is a change in the composition of the intent labels. This paper focuses on topic modeling in dealing with two tasks: first, to find new intents from user messages that are not yet included in any previous intents; and second, to reorganize existing intents by analyzing the topic model generated. In the analysis, two possible changes in intent compositions are intent merging and splitting. The topic modeling approaches used in this research are LDA as a baseline, as well as state-of-the-art Top2Vec and BERTopic. The labeled datasets for evaluation are taken from one of the major e-commerce companies in Indonesia and four public datasets. We evaluate the topic models by using the metrics of topic coherence, diversity, and quality. Our results show that the BERTopic and Top2Vec topic models produced better evaluation scores than the LDA topic model. Additionally, we perform proportion threshold analysis for reorganizing the intents of the datasets.