Particle filter for 3D fingertips tracking from color and depth images with occlusion handling

This paper presents a fingertips tracking system based on particle filter using Microsoft Kinect. The tracking is performed in two separate modules: image-based 2D position tracking and calibrated 1D depth tracking. The two modules is then combined to represent full 3D fingertips tracking. The separation is aimed for faster tracking by putting as most processing as possible in 2D while 3D position is obtained by just incorporating available depth information. Moreover, the error in 2D tracking can be corrected using the information from depth tracking and vice-versa. The fingertips are obtained by 2D convex hull algorithm on binary images which is extracted after setting a depth threshold. This assumes that hand and fingers is the foremost part of object nearest to camera during interaction. To deal with partial occlusion, we devised a simple retrace algorithm based on linear time-series analysis of fingertips trajectory. To evaluate the effectiveness of the system we design several scenarios involving contactless user interaction for zooming and motion capture for 3D animation creation. The evaluation is measured in terms of tracking accuracy and overhead computation. The 2D+1D tracking is experimentally shown to be more robust to occlusion with reasonably little overhead computation time.