Systems and methods for detecting motion in compressed video are provided. Some methods can include parsing a stream of compressed video, obtaining macroblock size information from the parsed stream, computing factors derived from the macroblock size information, computing adaptive threshold values derived from relative frame characteristics of the compressed video, comparing the factors derived from the macroblock size information with the adaptive threshold values, and detecting motion based upon the comparing when at least one of the factors exceeds at least one of the adaptive threshold values. In some embodiments, detecting the motion can include performing spatio-temporal filtering on macroblocks in which the motion is detected or performing spatio-temporal filtering on at least one non-motion macroblock.