A method of improving depth sensor data in real-time based on scene sensor data is provided that includes aligning depth images of depth-scene image pairs generated by a depth-scene image sensor pair with corresponding scene images wherein, for each depth-scene image pair, the depth image is warped such that locations of depths in the depth image are aligned with locations of corresponding pixels in the scene image, and improving at least some of the aligned depth images based on image data from one or more of the scene images.