Real-time modification of video images of humans allows for the video to be modified so that an expression of a subject human may be changed. Customer service agents may have more successful interactions with customers if they display an appropriate facial expression, such as to provide a particular emotional state. By determining an appropriate facial expression, and any deviation from a customer service agent's current expression, a modification to the video image of the customer service agent may be determined and applied. As a result, agents may not have a facial expression that is best suited to successfully resolve a purpose of the interaction, to have the customer be presented with the best-suited facial expression.