A method for summarizing image content from video images received from a moving camera includes detecting foreground objects in the images, determining moving objects of interest from the foreground objects, tracking the moving objects, rating movements of the tracked objects, and generating a list of highly rated segments within the video images based on the ratings.