Replacing content of a surface in video is described. In one or more embodiments, a surface captured in a scene of video content is replaced with replacement content. As part of doing so, an indication of the surface to be replaced is received. The indication may be received as user input to select the surface for replacement from a frame of video content or as a pattern that can be used to search for the surface in frames of the video content. Once the surface is identified, it is tracked throughout the video content using motion tracking techniques. The motion tracking identifies frames of the video content that include the identified surface as well as locations of surface in the identified frames. Based on the identified frames and locations, the replacement content is incorporated into the video content to replace the surface.