This disclosure is directed to techniques for determining a picture type for each of a plurality of frames included in a video sequence based on cross-correlations between the frames. The cross-correlations include first order cross-correlations between image information within pairs of frames included in the video sequence and second order cross-correlations between pairs of the first order cross-correlations. The first order cross-correlations may be analyzed to detect video transitional effects between the frames. The first and second order cross-correlations may be comparatively analyzed to determine temporal similarities between the frames. Therefore, the correlation-based determination techniques determine picture types for the frames based on the video transitional effects and the temporal similarities. The correlation-based determination techniques may calculate the first order cross-correlations between images within pairs of frames, or between sets of subimages within pairs of frames that are then averaged over the subimages for each of the pairs of frames.