A computer system, method, and computer-readable product for automatically generating a hierarchy of concepts described in a video using semantic concept matching that assists persons in searching for specific types of content within videos. A video segment is obtained, with the segment having a distinct running time with discrete information related to one or more topics within its content. The discrete information of the video segment is then parsed out and related to a predetermined relevant topic, and then one or more attributes for weighting each discrete information relevant topic within the video segment are determined. A hierarchy of concepts of the video segments is then created, based upon, at least, the weighting of the information topic within the running time of the video segment based upon the presence of the one or more attributes. Other information related to the video segment can also be used to modify the hierarchy.