The present disclosure describes a system and method for background identification for videos with large foreground objects. The method includes dividing a video clip into a multiple short video clips, each of which overlaps with neighboring short video clips of the multiple short video clips, detecting, in each short video clip, multiple background motion candidates, linking the background motion candidates between the neighboring short video clips to form multiple linked groups of background motion candidates, and identifying a linked group of background motion candidates that is most dominant among the multiple linked groups of background motion candidates.