A method and apparatus for generating video packets and a method and apparatus for restoring video are provided. The method of generating video packets includes: generating at least one pixel block consisting of at least one reference pixel and a plurality of neighboring pixels adjacent to the at least one reference pixel from pixels of a video frame; replacing pixel values of the neighboring pixels in the at least one pixel block with pixel difference values, which are difference values between a pixel value of one of pixels adjacent to the neighboring pixels and the pixel values of the neighboring pixels; generating packets so that a pixel value of the at least one reference pixel and the pixel difference values of the neighboring pixels are allocated to different packets according to positions of the pixels; and performing entropy coding of some of the packets including the pixel difference values of the neighboring pixels.