A route is selected for travel by an autonomous vehicle based on at least a level of difficulty of traversing the driving environment along that route. Vehicle signals, provided by one or more autonomous vehicles, indicating a difficulty associated with traveling a portion of a route are collected and used to predict a most favorable driving route for a given time. The signals may indicate a probability of disengaging from autonomous driving mode, a probability of being stuck for an unduly long time, traffic density, etc. A difficulty score may be computed for each road segment of a route, and then the scores of all of the road segments of the route are added together. The scores are based on number of previous disengagements, previous requests for remote assistance, unprotected left or right turns, whether parts of the driving area are occluded, etc. The difficulty score is used to compute a cost for a particular route, which may be compared to costs computed for other possible routes. Based on such information, a route may be selected.