In expansion and reduction of the size of a macroblock in a moving image encoding, a compression ratio falls because an information amount for describing size information of the macroblock increases. An image decoding device includes a CU dividing unit and a CU-size storing unit. The CU-size storing unit has stored therein size information of an encoded CU in an encoding target picture to be stored and size information of all CUs in preceding and following pictures used for prediction. The CU dividing unit predicts a CU size in an encoding target position using the CU size information stored in the CU-size storing unit and determines a CU size of a final encoding target CU.