An optical-see-through head mounted display (HMD) system is provided. The optical-see-through HMD system has a camera for generating image frames, a display device and a processor. The processor proceeds an interactive operation on each image frame. In the interactive operation, an image analysis is performed on the image frame to obtain positioning information of a marker and 3-dimensional information of an input device. According to the positioning information, the 3-dimensional information and eye position of an user, an image shielding process is performed to correct a portion of the frame to be displayed which is corresponding to the input device and a collision test is performed according to the positioning information and the 3-dimensional information of an input device to determine whether the input device touches the virtual image displayed by HMD. Then, an event corresponding to the touch position of the virtual image is executed.