Rendering an avatar in a selected environment may include determining as inputs into an inferred shading network, an expression geometry to be represented by an avatar, head pose, and camera angle, along with a lighting representation for the selected environment. The inferred shading network may then generate a texture of a face to be utilized in rendering the avatar. The lighting representation may be obtained as lighting latent variables which are obtained from an environment autoencoder trained on environment images with various lighting conditions.