A video display control system (30) automatically controls a video display responsive to detection of letterbox format input video signals having varying format display ratios. A detection circuit continuously detects the first and last lines of active video in a video signal. A memory stores scan line numbers corresponding to the first and last lines of active video. The height of the picture is determined from the scan line numbers of the first and last lines of active video. The picture height is indicative of the format display ratio of the letterbox input. A comparator circuit compares the picture height to a threshold corresponding to the widest expected format display ratio of a letterbox input signal. A control circuit, for example a microprocessor (30), is operable in a first mode of operation for enabling the memory to continuously update the stored scan line numbers unless the picture height exceeds the threshold.