A position and orientation measurement apparatus for measuring the position and orientation of an image capturing apparatus, which captures an image of a measurement object, relative to the measurement object, extracts configuration planes of the measurement object based on three-dimensional model data of the measurement object, and extracts measurement line segments to be used in detection of edges of a captured image from line segments which form the configuration planes. The position and orientation measurement apparatus projects the extracted measurement line segments onto the captured image based on an estimated position and orientation of the image capturing apparatus, selects visible measurement line segments which are not hidden by the extracted configuration planes, and calculates the position and orientation of the image capturing apparatus relative to the measurement object based on the visible measurement line segments and corresponding edges of the captured image.