The invention discloses a pole image collection system and method based on visual servo, and the method comprises the steps: (1), obtaining the video information of a pole through employing a PTZ camera according to a routing inspection requirement, and cutting a real-time image frame from the video; (2), recognizing and positioning the position of the pole in the cut real-time image; (3), judging whether the pole is located at the center of the cut real-time image or not: switching to step (5) if the pole is located at the center of the cut real-time image, or else, determining the deviation of the pole relative to the center of the cut real-time image; (4), determining the rotating direction of the PTZ camera through the position of the pole in the image, determining the rotating amount of the PTZ camera according to the deviation obtained at step (3), obtaining the real-time image of the pole at the current position again after the PTZ camera is adjusted, and switching to step (2); (5), adjusting the position of the PTZ camera from one corner of the pole in an S-shaped path, and carrying out the collection of the image of the pole through the above PTZ camera.