An initialization technique that may, for example, be used in an adaptive reconstruction algorithm implemented by structure from motion (SFM) techniques. The initialization technique computes an initial reconstruction from a subset of frames in an image sequence. The initialization technique may be performed to determine and reconstruct a set of initial keyframes covering a portion of the image sequence according to the point trajectories. In the initialization technique, a set of temporally spaced keyframe candidates is determined and two initial keyframes are selected from the set of keyframe candidates. The two initial keyframes are reconstructed, and then one or more additional keyframes between the two initial keyframes are selected and reconstructed.