Disclosed embodiments relate to a smart video conference environment. A live video of a conference participant can be segmented into a foreground, including the participant, and a background that excludes the participant. The background can be replaced by a virtual background determined based on one or more factors. The factors can include context data acquired from a calendar, content based on the context data, and transaction history. Further, interaction amongst conference participants can be monitored and utilized to update the background in one instance based on topics of conversation. Interactive content presented in the virtual background permits interaction by participants with the background. Furthermore, content can also be overlaid on the participant and the background in one instance.