An image encoder configured to process a Bayer image generated by passing through a color filter of a Bayer pattern includes: a detector configured to read the Bayer image in units of blocks and search for, in the blocks, a target pixel to be compressed and a plurality of candidate pixels which are located adjacent to the target pixel; a flag generator configured to compare a first pixel value of the target pixel with second pixel values based on pixel values of the plurality of candidate pixels, identify a reference pixel based on a comparison result, and generate a flag indicating relative direction information between the target pixel and the reference pixel; and a compressor configured to encode information corresponding to a comparison method applied by the flag generator and the comparison result and output the encoded information as a bitstream together with the flag.