Embodiments of the present invention provide systems, methods, and computer storage media for automatically synthesizing a content-aware fill using similarity transformed patches. A user interface receives a user-specified hole and a user-specified sampling region, both of which may be stored in a constraint mask. A brush tool can be used to interactively brush the sampling region and modify the constraint mask. The mask is passed to a patch-based synthesizer configured to synthesize the fill using similarity transformed patches sampled from the sampling region. Fill properties such as similarity transform parameters can be set to control the manner in which the fill is synthesized. A live preview can be provided with gradual updates of the synthesized fill prior to completion. Once a fill has been synthesized, the user interface presents the original image, replacing the hole with the synthesized fill.