An automatic object tracking system is configured to perform an automatic object tracking method. A horizontal coordinate axis and a vertical coordinate axis are defined in image capturing data captured by a handheld device. When a feature object is recognized, a sampling frame is defined to surround the feature object, and a reference point is defined within the sampling frame. A horizontal distance and a vertical distance from the reference point to a target coordinate is calculated. When the horizontal distance is larger than a horizontal threshold, a turning command is issued to move the reference point towards the target coordinate horizontally by at least one horizontal unit. When the vertical distance is larger than a vertical threshold, the turning command is issued to move the reference point in the image capturing data towards the target coordinate vertically by at least one vertical unit.