Motions and gestures can be detected using a video capture element of a computing device even when the video capture element is not able to accurately capture the motion. Information about the background in the image information can be determined, and the way in which that background information is occluded can be used to determine the motion. In at least some embodiments, edges are detected in the video information. Images of foreground objects can then be isolated from edges of background images by comparing histograms of multiple frames of video. The remaining data is indicative of a direction and speed of motion, which can be used to infer a determined gesture even though that gesture was not visible in the captured video information.