A method, computer system, and computer program product are provided for virtual background replacement during a video communication session. A frame comprising a user image and a captured background is obtained from a video capture device as part of a video stream. The frame is processed to replace the captured background with a virtual background. A change to an exposure characteristic of the video capture device is determined. In response to determining the change to the exposure characteristic, the virtual background is modified to match a virtual background brightness with a user image brightness to produce a modified virtual background. The frame is modified by combining the user image with the modified virtual background.