Systems and techniques are provided for generating depth information from one or more images. For instance, a method can include obtaining an image of a scene and obtaining depth information associated with one or more objects in the scene. The method can include processing, using an encoder of a neural network model, the image and the depth information to generate a feature representation of the image and the depth information. The method can further include processing, using a decoder of the neural network model, the feature representation of the image and the depth information to generate a depth output corresponding to the image.