An encoder segments frames of video data and associates metadata with segments. The metadata elements can be associated with the segments that include areas of the frame associated with the metadata elements. A motion matcher can match segments of a reference frame to pixels of a current nonkey frame being encoded when a metadata associator associates elements of metadata with segments of the segmentation of the reference frame and associates a matched segment's metadata elements with matched areas of the current frame. The metadata elements might represent actions to be taken when a user of the video data indicates a selection of an area of an image that has that metadata element associated therewith. The metadata associations can be included in the encoded video data or deduced by a decoder. The metadata associations can be independent of segment indices or other segment changes.