A computer-implemented method for tracking with visual object constraints includes receiving a lingual constraint and a video. A word embedding is generated based on the lingual constraint. A set of features is extracted for one or more frames of the video. The word embedding is cross-correlated to the set of features for the one or more frames of the video. A prediction indicating whether the lingual constraint is in the one or more frames of the video is generated based on the cross-correlation.