A face model providing portion provides an stored average face model to an estimation portion estimating an affine parameter for obtaining a head pose. An individual face model learning portion obtains a result of tracking feature points by the estimation portion and learns an individual face model. The individual face model learning portion terminates the learning when a free energy of the individual face model is over a free energy of the average face model, and switches a face model provided to the estimation portion from the average face model to the individual face model. While learning the individual face mode, an observation matrix is factorized using a reliability matrix showing reliability of each observation value forming the observation matrix with emphasis on the feature point having higher reliability.