The present invention discloses a method for compressing and decompressing pixel data and the device thereof. The method comprises receiving current pixel data from a preprocessor and comparing the current pixel data with predictor pixel data to obtain a section index from a quantization table. Output the section index to a memory as compressed data. Then, receive the compressed data from the memory. Use the quantization table to look up a representative level corresponding to the compressed data and output decompressed pixel data based on the representative level to an image processing unit. The device mentioned above comprises a memory, a compressor and a decompressor. The memory stores compressed data. The compressor quantizes and compresses pixel data and outputs the compressed pixel data to the memory. The decompressor receives the compressed pixel data and decompresses the compressed pixel data and outputs the decompressed pixel data as pixel data.