Comparing Classical Distance Measures and Word Embeddings for Automatic Short Answer Grading

In the educational process, students' answers to essay questions are one of the cognitive methods to measure students' understanding of a topic being studied. But checking essay answers is certainly much more difficult than multiple-choice answers. Apart from absorbing much energy and time, it may also be biased depending on the human rater's subjectivity. To overcome this, researchers have already started to develop Automatic Short Answer Grading (ASAG) by exploring the field of natural language processing (NLP). However ASAG research specifically for Indonesian is still limited. This research aims to find the right method to improve the accuracy of the ASAG system in Indonesia focusing on the computer science domain, with a deep learning approach. We started our research by combining the feature extraction method Bag of Words and Term Frequency-Inverse Document Frequency (TF-IDF) with linear regression, Support Vector Regression, and Random Forest Regression. And then employing BERT and FastText. Both experiments produced similar performance, with an F1-score on average of 0, 72, which is categorized as low similarity. This opens opportunities for further research in word embeddings, especially the transformer method that becomes state-of-the-art.