A method of decoding an image including extracting information that indicates an intra prediction mode applied to a current block to be decoded, from a bitstream; determining one of neighboring pixels adjacent to the current block and filtered neighboring pixels filtered from the neighboring pixels as reference pixels, based on at least one of a size of the current block and an intra prediction mode of the current block; performing intra prediction on the current block using the extracted information and the determined reference pixels, wherein, when the current block has a size of N×N, where N is integer, the neighboring pixels include 2N neighboring pixels adjacent to an upper side and an upper right side of the current block and 2N neighboring pixels adjacent to a left side and a below left side of the current block.