Systems, methods, and computer-readable media are described for performing automated analysis of frame-over-frame pixel changes in recorded video of the display output of a computing device to determine whether the computing device presented an animated buffering indicator or other animated problem indicator. A system may be configured to detect motion by determining frame-over-frame pixel intensity changes at various pixel locations across a number of frames, then to determine whether the pixel locations that suggest motion (such as those pixel locations that had sufficient intensity change when accounting for potential noise in the video data) are concentrated in an area of the screen in which problem indicators are expected to be displayed. The system may then determine whether the shape of the pixel locations that indicated sufficient motion match an expected shape or path of motion for a given class of problem indictors.