Video encoding and decoding is implemented with auto encoders using luminance information to derive motion information for chrominance prediction. In one embodiment YUV 4:2:0 video is encoded and decoded in which luminance information is downsampled to generate predictions from chrominance components of a reference frame. In a related embodiment, more than one reference frame is used for predictions. In another embodiment, convolutions and transpose convolutions implement derivation of motion information.