A method of processing a depth image includes receiving a high-resolution color image and a low-resolution depth image corresponding to the high-resolution color image, generating a feature vector based on a depth distribution of the low-resolution depth image, selecting a filter to upsample the low-resolution depth image by classifying a generated feature vector according to a previously learnt classifier, upsampling the low-resolution depth image by using a selected filter, and outputting an upsampled high-resolution depth image.