A video signal follow-up processing system for adaptively tracking to the moving of a subject. A detection feature pattern is formed through acquisition of brightness and hue frequency characteristic data based on pixel information in a detection measurement frame, a similarity calculation method, which can distinguish a reference measurement frame from other areas, is selected on the screen, and the position of a detection measurement frame with a feature pattern having the highest similarity with the standard feature pattern obtained from the reference measurement frame is determined, in order to change and control an image to be projected on the display screen based on the positional information of the detection measurement frame, so that the video signal follow-up processing system can adaptively track to the moving of the subject.