Visual object tracking is the process of continuously localizing visual object in a video sequence. We would like to investigate the problem of short-term model-free tracking which the main purpose is to track any object just based on an annotation box of object. Many factors affect the performance of the tracking algorithm. In the Visual Tracker Benchmark, there are eleven challenges in object tracking. There has not been a single tracker that successfully handles all of these scenarios. In addition, the tracker must be fast enough to be useful in real applications. We propose a new tracking algorithm within the Bayesian framework. The proposed algorithm is constructed by solving optimally particle filters (OPF) efficiently using spectral methods. Therefore, the constructed tracker is called as Spectral Tracker (ST). Although ST can efficiently compute object position, it cannot estimate the scale and rotation directly. To overcome this weakness, it is proposed to use multiple observation points simultaneously and to use information on the observation point movement to estimate scale and rotation. In the experiments, the performance of ST tracker was compared with 9 relevant trackers based on 100 data sets. The experimental results on on tracker performance show that increasing performance especially in tracker precision and success rate.