A video signal processor recursively removes noise from luminance signal data, storing only one frame of the luminance signal data, and detects luminance motion by comparing the current and stored luminance data to decide whether to output the current color difference data, the average of the current color difference data and the color difference data one frame before, or the average of the current color difference data and the color difference data two frames before. The decision may also involve detection of color noise by comparison of the color difference data with the data one frame before, or detection of color motion by comparison of the color difference data with data two or four frames before. Color data stored for color noise detection may also be used for recursive color noise removal. Accurate noise reduction is possible with a comparatively small amount of memory.