A digital time base corrector reduces the capacity of a memory to approximately a 1 H of image data while maintains the interleave relationship between a luminance signal and a chrominance signal of a video signal. Polarity data indicating the polarity of a burst signal is generated every 1 H, and a latch circuit is provided for holding and outputting the polarity data during a period of reading image data and polarity data from an image data memory and a sub-carrier phase memory, corresponding to the 1 H in which the polarity data was generated. When dropout occurs in a video signal, the image data and the polarity data are inhibited from being written into the image data memory and the sub-carrier phase memory. When discrepancy is detected between read polarity data and polarity data held in the latch circuit, the color phase of the read image data is inverted.