Nowadays, the social intelligent machine is developed to be more human-like by understanding human emotion. Therefore, many researches attempt to capture nonverbal signals which visually appears on facial expressions to recognize human emotions. Face is being explored because face mostly reveals the emotion contains in it. Many researches on facial expression recognition have been done and delivered optimistic results, however continuous research must be afforded to increase the performance of some methods. This paper is written as a part of doctoral research study on human emotion recognition as an effort to depict and elaborate the roadmap of research on automatic emotion recognition. In this paper, firstly we describe about Social Signal Processing as the emotion recognition domain. Secondly, the analysis of a human emotion model. Thirdly, the method of Facial Expression Recognition (FER), followed by the description about state of the art research on FER. Lastly, we explore about the prospect of the next research on FER as a part of the development on this field.