A method of converting a two-dimensional video to a three-dimensional video, the method comprising: comparing an image of an nth frame with an accumulated image until an n−1th frame in the two-dimensional video to calculate a difference in a color value for each pixel; generating a difference image including information on a change in a color value for each pixel of the nth frame; storing an accumulated image until the nth frame by accumulating the information on the change in the color value for each pixel until the nth frame; performing an operation for a pixel in which a change in a color value is equal to or larger than a predetermined level by using the difference image to generate a division image and a depth map image; and converting the image of the nth frame to a three-dimensional image by using the depth map image.