An apparatus and method for estimating disparity based on a visibility energy model includes an energy calculator to calculate energy related to stereo matching of each of a left image and a right image which constitute a stereo image, a map generator to generate a visibility map for determining an error in disparity between the left image and the right image using the energy, an energy recalculator to recalculate energy with respect to a region including a visibility error generated due to the error in disparity in the visibility map, and a disparity determiner to determine disparity from the stereo image using recalculated energy of each of the left image and the right image.