An optical tracking device includes a light source, an image sensor and a processing unit. The light source emits light at a lighting frequency. The image sensor outputs an operating image when the light source is being turned on and outputs a background image when the light source is being turned off. The processing unit is configured to obtain background information from the background image, calculate a differential image of the operating image and the background image, obtain object information from the differential image and compare the background information and the object information thereby removing background noise.