A method, apparatus, and system with tracking are disclosed. The apparatus is configured to acquire a feature map of a template image, a color histogram of a foreground of the template image that has an object, and a color histogram of a background of the template image other than the foreground, acquire a feature score map, and a bounding box map corresponding to the feature score map, based on the feature map of the template image and a feature map of a search image, acquire a color score map based on the color histogram of the foreground, the color histogram of the background, and a color value of the search image, acquire a final score map, and a bounding box map corresponding to the final score map, based on the feature score map, the color score map, and the bounding box map, and output the corresponding bounding box.