A method is provided for Cross Video Temporal Difference (CVTD) learning. The method adapts a source domain video to a target domain video using a CVTD loss. The source domain video is annotated, and the target domain video is unannotated. The CVTD loss is computed by quantizing clips derived from the source and target domain videos by dividing the source domain video into source domain clips and the target domain video into target domain clips. The CVTD loss is further computed by sampling two clips from each of the source domain clips and the target domain clips to obtain four sampled clips including a first source domain clip, a second source domain clip, a first target domain clip, and a second target domain clip. The CVTD loss is computed as |(second source domain clip−first source domain clip)−(second target domain clip−first target domain clip)|.