Energy saving is an effort to decrease and minimize unnecessary energy consumption. The energy saving and energy efficiency is one of the famous issues in the last decade since the energy resource is rapidly depleted. Reducing unnecessary energy consumption could be performed by architectural design or automatic system approach. This paper proposes an idea to develop a model of energy usage in a room by detecting and counting the number of peoples and detecting their activities. In order to achieve the energy consumption model, the method consists of two modules: the human detection and the activity quantization of each detected object. The human is detected using multiple registered image channels that computed in linear and non-linear transformations of the input image. The activity of each detected object is approached by computing the movement of an object in a certain time. Moreover, the human activities were represented based on local and global motion of all detected object. Videos were recorded in a classroom to evaluate the performance of the method. In evaluation, the cooling load weight model of student walking and pointing hand activities were computed in certain schema. The results show that the method can predict human activity effectively. The proposed technique is suitable for a small cost of the room energy controller and achieves the goal of energy saving.