A lens driving device includes a control unit configured to control movement of a first lens to perform zooming operation and a second lens to perform focus adjustment based on information of a position of the first lens and a position of the second lens stored for each object distance, and when the first lens is on a telephoto side of a predetermined position, the control unit limits movement of the second lens to a position corresponding to a second object distance closer than the first object distance, and if a predetermined condition is satisfied after the second lens has reached a predetermined region including the position corresponding to the first object distance, the control unit permits the movement of the second lens to the position corresponding to the second object distance.