A system for interactively organizing and browsing video automatically processes video, creating a video table of contents (VTOC), while providing easy-to-use interfaces for verification, correction, and augmentation of the automatically extracted video structure. Shot detection, shot grouping and VTOC generation are automatically determined without making restrictive assumptions about the structure or content of the video. A nonstationary time series model of difference metrics is used for shot boundary detention. Color and edge similarities are used for shot grouping. Observation about the structure of a wide class of videos are used for the generating the table of contents. The use of automatic processing in conjuction with input from the user provides a meaningful video organization.