What is disclosed is a system and method for enhancing a spatio-temporal resolution of a depth data stream. In one embodiment, time-sequential reflectance frames and time-sequential depth frames of a scene are received. If a temporal resolution of the reflectance frames is greater than the depth frames then a new depth frame is generated based on correlations determined between motion patterns in the sequence of reflectance frames and the sequence of depth frames. The new depth frame is inserted into the sequence of depth frames at a selected time point. If a spatial resolution of the reflectance frames is greater than the depth frames then the spatial resolution of a selected depth frame is enhanced by generating new pixel depth values which are added to the selected depth frame. The spatially enhanced depth frame is then inserted back into the sequence of depth frames.