A method of tracking a moving object includes measuring displacement of an object to be tracked, obtaining a particle of the object to be tracked using the measured displacement, and tracking the object using pose information of the object in an image thereof and the obtained particle. A control apparatus includes an imaging module to perform imaging of an object and generates an image, and a tracking unit to acquire displacement and pose information of the object using the generated image of the object, to set a particle of the object using the acquired displacement of the object, and to track the object using the pose information of the object and the particle.