A technology is described for performing structure from motion for unordered images of a scene with multiple object instances. An example method can include obtaining a pairwise match graph using interest point detection for obtaining interest points in images of the scene to identify pairwise image matches using the interest points. Multiple metric two-view and three-view partial reconstructions can be estimated by performing independent structure from motion computation on a plurality of match-pairs and match-triplets selected from the pairwise match graph. Pairwise image matches can be classified into correct matches and erroneous matches using expectation maximization to generate geometrically consistent match labeling hypotheses and a scoring function to evaluate the match labeling hypotheses. A structure from motion computation can then be performed on the subset of match pairs which have been inferred as correct.