Because neighboring frames may affect how a current frame is perceived, we examine different neighborhoods of the current frame and select a neighborhood that impacts the perceived temporal distortion (i.e., when frames are viewed continuously) of the current frame most significantly. Based on spatial distortion (i.e., when a frame is viewed independently of other frames in a video sequence) of frames in the selected neighborhood, we can estimate initial temporal distortion. To refine the initial temporal distortion, we also consider the distribution of distortion in the selected neighborhood, for example, the distance between the current frame and a closest frame with large distortion, or whether distortion occurs in consecutive frames.