A system and a method of detecting abnormal or security-significant segments of a video are provided. The video is divided into several segments. A set of features of each segment of the video is extracted in order to calculate a set of factors corresponding to each segment. A value deemed abnormal of each segment is calculated according to the set of factors corresponding to each segment. One or more abnormal segments are determined from the segments based on the abnormal values. The set of features includes a color variable feature, a movement variable feature, a movement variable ratio feature and a background variable feature.