A system and method for generating robust depth maps includes a depth estimator that creates a depth map pyramid structure that includes a plurality of depth map levels that each have different resolution characteristics. In one embodiment, the depth map levels include a fine-scale depth map, a medium-scale depth map, and a coarse scale depth map. The depth estimator evaluates depth values from the fine-scale depth map by utilizing fine-scale confidence features, and evaluates depth values from the medium-scale depth map and the coarse-scale depth map by utilizing coarse-scale confidence features. The depth estimator then fuses optimal depth values from the different depth map levels into an optimal depth map.