A semantic image inference method and device are disclosed. Provided are a semantic image inference method and device which employ a technology for generating an empty area to which an image is to be extended, by using an inpainting technique and a segmentation map having an extended empty area, wherein, since there is no information in an empty area of an image to be extended, a segmentation map having an extended empty area is first generated on the basis of a segmentation map from an input image and the empty area of the image to be extended is then filled with information on the basis of the input image and the segmentation map having the extended empty area.