Provided is a video encoding method including: calculating a difference value between a rate-distortion (R-D) cost of a first coefficient level from among coefficient levels of a sub-block and an R-D cost of a second coefficient level adjacent to the first coefficient level by using a distortion difference calculated by using an integer of the first coefficient level and an integer of the second coefficient level and a rate variation between the first coefficient level and the second coefficient level; selecting one of the integer values of the first and second coefficient levels as a modified value of the first coefficient level based on the difference value; and outputting the modified value of the first coefficient level.