Provided is a method of removing noise from a sparse depth map. The method includes setting, as nodes, sample points in a sparse depth map including depth information at a certain position of an image plane, constructing a mesh by connecting the set nodes, and removing noise from the meshed node, based on connection information between the meshed node and neighboring nodes adjacent thereto and color information about a color image corresponding to the sparse depth map. A depth value is allocated to each of the sample points.