In a video encoding apparatus of the invention, a video camera is provided with a moving direction detecting means for detecting movement of the camera in preset moving direction and outputting a detecting signal, a movement information generator is installed for operating the displacement amount of the video camera based on the detecting signal outputted from the moving direction detecting means and calculating the specific position on the memory corresponding to the operated displacement amount of the video camera and outputting the position to a reference signal generator, and plural blocks stored to the specific position on the frame memory are read based on output from the movement information generator.