The controller system for quadrotor is a challenging research topic due to various application purposes. The usually used controller for quadrotor is developed based on a PID method due to its low cost, simple structure and easy to design with an acceptable error. However, determining precisely the PID's parameters is very difficult, especially when the characteristics of the plant are highly nonlinear, underactuated and cross-coupling. In this paper, a neural networks-based controller system through Elman recurrent learning mechanism is then proposed. Using a real-time flight dataset from the quadrotor, neural networks based direct inverse control scheme of attitude and altitude control system is constructed and tested through simulation. Experimental results show that the Elman neural networks-based controller system works within very low MSSE when it is used to follow the reference flight testing dataset. Experiments also confirmed that the Elman recurrent neural network shows better performance characteristics compared with that of the Backpropagation neural network.