Provided is a video encoding apparatus, including a signal separator to separate a differential image block into a first domain and a second domain, based on a boundary line included in the differential image block, the differential image block indicating a difference between an original image and a prediction image with respect to the original image, a transform encoder to perform a transform encoding with respect to the first domain using a discrete cosine transform (DCT), a quantization unit to quantize an output of the transform encoding unit in a frequency domain, a space domain quantization unit to quantize the second domain in a space domain, and an entropy encoder to perform an entropy encoding using outputs of the quantization unit and the space domain quantization unit.