A vision system for a vehicle includes a camera and a control. The control determines information on traffic signs and determines whether an indicated speed limit is for the lane being traveled by the vehicle. The vision system determines whether the indicated speed limit is for the lane being traveled by the vehicle responsive to a determination that the sign is at the left side of the lane and has an indicator representative of the right side of the lane and leaves the field of view at its left side, determination that the sign is at the right side of the lane and has an indicator representative of the left side of the lane and leaves the field of view at its right side, or determination of a speed limit sign at both sides of the road being traveled by the vehicle with both signs indicating the same speed limit.