The video coding device calculates a first value representing energy of the prediction error signals of the block to be processed, eliminates specified transform coefficients when at least one of a width and a height of the block to be processed exceeds a predetermined value and calculates a second value representing energy of the transform coefficients other than the eliminated transform coefficients, and decides not to make the candidate of prediction mode the optimal prediction mode, without executing quantizing, arithmetic-coding and local decoding, in a case where the second value is smaller than a predetermined degree as compared to the first value.