In an exemplary embodiment, in order to avoid enlargement of a circuit scale to perform coding processing in real time, a block size of a PU (evaluation PU (Prediction Unit)) used to evaluate a merge mode is restricted and the number of evaluation target merge candidates is restricted. In the case where the evaluation PU has the block size in which a CU (Cording Unit) is divided, at least one merge candidate common to a merge candidate list of a PU (inclusion PU) having the block size including the evaluation PU and a merge candidate list of each of a plurality of evaluation PUs is selected as the evaluation target merge candidate from the merge candidate lists of the evaluation PUs. In performing cost evaluation processing for the evaluation PU, cost evaluation processing is simultaneously performed on the inclusion PU using a calculated prediction residual.