The image decoding apparatus and the image decoding method according to one aspect of the present invention have a configuration for storing an image decoded in the past as a reference picture into a frame memory, in a field structure in which top lines in the reference picture are stored in a top area and bottom lines in the reference picture are stored in a bottom area, in order to use a part of the image decoded in the past as a reference block in a picture being presently decoded; and selectively copying and storing an uppermost top line or an uppermost bottom line in the reference picture to areas on the uppermost top line and the uppermost bottom line in the reference picture in the top area and the bottom area in the frame memory, and selectively copying and storing a lowermost top line or a lowermost bottom line in the reference picture to areas under the lowermost top line and the lowermost bottom line in the reference picture in the top area and the bottom area in the frame memory.