An image processing device comprises an image acquisition apparatus configured to acquire an image of an object, a feature extraction unit configured to extract a feature portion from a visual field of the image acquired by the image acquisition apparatus, a weighting section configured to calculate a weight in accordance with extraction conditions of the feature portion extracted by the feature extraction unit to weight the feature portion, and a position orientation obtaining section configured to obtain a position and an orientation of the image acquisition apparatus on a reference coordinate system from the feature portion weighted by the weighting section.