In a video decoding device, a quantization step size decoding unit calculates a quantization step size that controls a granularity of the inverse quantization by, based on an image prediction, selectively using a mean value of at least a quantization step size assigned to a leftwardly adjacent neighboring image block already decoded and a quantization step size assigned to a upwardly adjacent neighboring image block already decoded or a quantization step size assigned to an image block decoded immediately before.