A compressed dynamic image encoding device is provided, in which a motion vector is generated by searching a reference image read from a frame memory for an image area most similar to an image area of a video input signal; a motion-compensated reference image is generated from the motion vector and the reference image read from the frame memory; a prediction residual is generated, by subtracting the motion-compensated reference image from the video input signal; the reference image to be stored in the frame memory is generated, by adding the motion-compensated reference image and the result of processing of orthogonal transform, quantization, inverse quantization, and inverse orthogonal transform performed to the prediction residual; and an encoded video output signal is generated by the processing of orthogonal transform, quantization, and variable-length encoding performed to the prediction residual. The reference image comprises on-screen reference images A, B, and C, located inside a video display screen, and an off-screen reference image D located outside the video display screen, and the off-screen reference image D is generated based on the positional relationship of plural similar reference images A and B of the on-screen reference images A, B, and C.