A method for detecting an emotional state of a user includes obtaining a first data stream indicative of facial appearance and gaze direction of the user as the user is viewing a scene, determining, based on the first data stream, facial expression feature information indicative of emotional facial expression of the user, obtaining a second data stream indicative of visual content in a field of view of the user, determining, based on the second data stream, visual feature information indicative of visual content in the scene, determining emotional state information based on analyzing the facial expression feature information determined based on the first data stream and the visual feature information determined based on the second data stream, and performing an operation with respect to the emotional state information, wherein the emotional state information is indicative of the emotional state of the user.