Aspects of the disclosure relate to controlling a first vehicle in an autonomous driving mode. While doing so, a second vehicle may be identified. Geometry for a future trajectory of the first vehicle may be identified, and an initial allowable discomfort value may be identified. Determining a speed profile for the geometry that meets the value may be attempted by determining a discomfort value for the speed profile based on a set of factors relating to at least discomfort of a passenger of the first vehicle and discomfort of a passenger of the second vehicle. When a speed profile that meets the value cannot be determined, the value may be adjusted until a speed profile that meets the value is determined. The speed profile that meets an adjusted value is used to control the first vehicle in the autonomous driving mode.