A depth map aligning method and a system are provided. The depth map aligning method includes the following steps. A view warping is performed to a left-view image and a right-view image of a real-view image according to a corresponding depth map to obtain a right-virtual view image, a left hole information, and a right hole information thereof. A residual difference of pixel, which is not a hole, is obtained by pixel to pixel. If the residual difference of the pixel is larger than a first threshold value, a reverse warping is performed to obtain a coordinate of the pixel. The depth value of the pixel is aligned according to the depth value of one or more pixels in a vicinity of the obtained coordinate.