The number of user reviews for a mobile app can reach thousands so it will take a lot of time for app developers to sort through and find information that is important for further app development. Therefore, this study aims to automatically classify mobile application user reviews. Automatic classification conducted in this study is using machine learning approach. The features extracted from user review are unigram, bigram, star rating, review length, as well as the ratio of the number of words with positive and negative sentiment. For classification algorithms, we used Naïve Bayes, Support Vector Machine, Logistic Regression and Decision Tree. The experiment result shows that Logistic Regression gives the best F-Measure of 85% when combined with unigram plus sentence length and sentiment score. Unigram was proven as the most important feature since the additional features like sentence length and sentiment score only increased the F-measure around 1%. Bigram and star rating has negative impact on the classifier performance.