A video signal recording and playback apparatus for recording each field of a video signal as a plurality of segments on a plurality of tracks, including a circuit for inserting segment index signals after conversion of the video signal to a recording signal, to be recorded together with the recording signal to designate the respective segments. During varied-speed playback operation, the segment index signals are detected in the playback signal, and the data thus obtained are used to control the operation of a memory which temporarily stores successively produced portions of the playback signal. A playback signal can thereby be obtained by combining output signals from the memory, which will produce a satisfactory display image during varied-speed playback operation, e.g. during high-speed playback, with only simple signal processing being required.