The subject disclosure relates to generating models from images. In an aspect, multi-view semantic segmentation is provided to recognize and segment images at the pixel level into semantically meaningful areas, and which can provide labels with a specific object class. In further aspects, a partition scheme is provided that can separate objects into independent blocks using major line structures of a scene. In addition, an inverse patch-based orthographic composition and structure analysis on a block is provided that can regularize noisy and missing reconstructed 3D data to facilitate image-based modeling.