The technology relates to determining general weather conditions affecting the roadway around a vehicle, and how such conditions may impact driving and route planning for the vehicle when operating in an autonomous mode. For instance, the on-board sensor system may detect whether the road is generally icy as opposed to a small ice patch on a specific portion of the road surface. The system may also evaluate specific driving actions taken by the vehicle and/or other nearby vehicles. Based on such information, the vehicle's control system is able to use the resultant information to select an appropriate braking level or braking strategy. As a result, the system can detect and respond to different levels of adverse weather conditions. The on-board computer system may share road condition information with nearby vehicles and with remote assistance, so that it may be employed with broader fleet planning operations.