Embodiments are directed towards providing supplemental content to a viewer based on the viewer's environment while the viewer is viewing content. Content is provided to a content receiver for presentation to the viewer. While the viewer is viewing the content, an image of the viewer's environment is captured and analyzed to detect the context of the environment. Supplemental content is selected based on this detected context and presented to the viewer, which allows the supplemental content to be tailored to the viewer. One or more additional images of the viewer's environment are captured while the viewer is viewing the supplemental content and the context of the viewer's environment emotion while the viewer is viewing the supplemental content is detected. This subsequent context is analyzed to determine if the viewer had an expected response to the supplemental content, which can be used to determine the efficiency of the supplemental content.