Given an input progressive sequence, a video encoder creates a dual-layer stream that combines a backwards-compatible interlaced video stream layer with an enhancement layer to reconstruct full-resolution progressive video. Given two consecutive frames in the input progressive sequence, vertical processing generates a top field-bottom field (TFBF) frame in a base layer (BL) TFBF sequence, and horizontal processing generates a side-by-side (SBS) frame in an enhancement layer (EL) SBS video sequence. The BL TFBF and the EL SBS sequences are compressed together to create a coded, backwards compatible output stream.