A method (and structure and computer product) of temporal action localization in video data includes receiving a stream of video data and determining all proposals in the video data stream, the proposals being candidate regions for temporal action in the video data stream. Values for a pair-wise relation function are calculated for relating the proposals, wherein the pair-wise relation function calculates a scalar value representing a pair-wise relation weight for pairs of the proposals.