Aspects of the disclosure provide for a method for identifying speed limit signs and controlling an autonomous vehicle in response to detected speed limit signs. The autonomous vehicle's computing devices identifies a speed limit sign in a vehicle's environment and a location and orientation corresponding to the speed limit sign. Then, the and orientation location of the speed limit sign is determined to not correspond to a pre-stored location and a pre-stored orientation of a speed limit sign that is pre-stored in map information. An effect zone of the speed limit sign is determined based on the location and orientation of the speed limit sign and characteristics of surrounding areas or other detected object before or after the speed limit sign. The autonomous vehicle's computing devices determines a response of the vehicle based on the determined effect zone, and controls the autonomous vehicle based on the determined response.