An apparatus for controlling object tracking and a method therefor are provided. The apparatus includes an object detector configured to detect an object in an image, an object tracker configured to track the object, a learning device configured to learn whether to enable the object detector based on features of the object and tracking results of the object tracker, and a controller configured to determine whether to enable the object detector by interworking with the learning device.