Provided is a building roof outline recognition device which facilitates automation. A small-building candidate region is extracted based on normalized DSM data using a region segmentation method. By using the building candidate region as a marker and further using height information included in the normalized DSM data and the like, a building region is extracted using a region growing method. While referencing the height information, from among edges appearing in the orthoimage, edges fitting with an outer shape of the building region are extracted as an outer shape of the roof when viewed from above. Further, for a building judged to have a slanted roof, a roof structure line which is the boundary between the roof surfaces is extracted from the building region, and a three-dimensional structure of the building roof outline is determined from the roof outer shape and the roof structure line.