Provided are an apparatus and method for generating panorama images and an apparatus and method for tracking an object using the same. The apparatus for generating panorama images estimates regional motion vectors with respect to respective lower regions set in an image frame, determines a frame motion vector by using the regional motion vectors based on a codebook storing sets of normalized regional motion vectors corresponding to a camera's motion and then accumulates motion mediating variables computed with respect to respective continuous image frames to determine an overlapping region, and matches the overlapping regions to generate a panorama image.