In one embodiment, a method implemented on a computing device for deriving timeline metadata for video content includes: capturing timeline elements through analysis of at least one of audio, visual or language aspects of the video content, interpreting the timeline elements according to associated inferences as indicative of timeline states, evaluating combinations of the timeline states according to a set of rules to derive timeline metadata, where the timeline metadata is associated with at least a portion of the video content.