Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating a depth map of a scene from a camera image using a neural network. One of the methods includes obtaining an image captured by a first sensor. A neural network processes the image to generate a respective score for each of a plurality of locations in the image. Known depth data specifying respective known depth values for some of the locations in the image is obtained. A depth output is generated that assigns a depth value to some of the locations in the image, including determining whether the score for a location exceeds a threshold; and when the score exceeds the threshold and the known depth value is available for the location, assigning the known depth value for the location to the location in the depth output.