In this paper, we present a technique for visual tracking in the field of Human Sperm motion. Application of sperm cell tracking is mainly important in Intracytoplasmic Sperm Injection (ICSI), a medical procedure that has enabled the In Vitro Fertilization (IVF) of a single sperm which is injected directly into an egg. In this paper, we consider the problem of tracking single object in video sequences of human sperms and a newly developed Smoothing Stochastic Approximate Monte Carlo (SSAMC) based tracker enhanced by Particle Swarm Optimization (PSO). The problem for this research is that the motility or movement of Human Sperm is fast and unpredictable. In addition, each and every sperms have closely similar size and shape. To solve this problem, we used PSO for searching algorithm (finding the best target) in a Search Window, it can reduce the search space in every each consecutive frame. The measurement results of the proposed method are then compared with the manual measurements done by experts. The experiment results were conducted on both open video data and our own video data. Experiment results showed that the proposed method can handle our specific problem in human sperm cell tracking, and give us a better result as compared to our previous tracker, which used geometric transition dynamic model and without any enhancement by PSO.