Aspects of the disclosure provide for identifying problematic areas within a service area for an autonomous vehicle transportation service. For instance, a starting location within the service area corresponding to a potential pickup location for passengers or cargo for the service may be identified. A destination within the service area may be identified. A simulation may be run in order to determine a route for a simulated vehicle to travel between the starting location and the destination. That the route includes a particular type of maneuver may be determined. A new simulation without allowing the simulated vehicle to complete the particular type of maneuver may be run. Whether the simulated vehicle reaches the destination in the new simulation may be determined. Based on the determination of whether the simulated vehicle reaches the destination in the new simulation, the starting location and destination location may be flagged as potentially problematic areas.