The specification relates to determining either or both of the pose of a camera with respect to a real object or a three dimensional structure of the real object. In some of the described embodiments, a captured image including the real object is analyzed with respect to a reference image that also includes the real object. A parameter set is estimated by using a homography between the reference image and the captured image. An update to the parameter set is made by using an iterative minimization process that includes comparing a first set of pixels in the reference image with a computed set of pixels in the captured image, where the computed set of pixels used for comparison varies in different iterations of the iterative minimization process.