The present invention provides an image signal processing apparatus and a method thereof in which each of the fields forming the unit-frame is specified, with respect to the inputted image signal, based on a difference value calculated in signal level between a detected pixel in a current field and a detected pixel at the same position in a field which comes one frame behind the current field, a motion vector for a field which comes two frames behind the current field is detected, with respect to the detected pixel in the current field, the detected pixel is shifted, with respect to the specified first field, in a direction opposite to the motion vector within the range of the detected motion vector, the detected pixel is shifted, with respect to the specified fourth field, in a direction along the motion vector, and the detected pixels is shifted, with respect to the specified second and third fields, so as to make the pixels gradually closer to the pixel position shifted with respect to the fourth field, in the consecutive order of the fields from the first field, in the direction along the motion vector or in the direction opposite to the motion vector.