The technology relates to planning trajectories for self-driving vehicles in order to transport passengers or cargo from a pickup location to a destination. Trajectory planning includes generating a speed plan for an upcoming portion of the trip in view of one or more constraints. The constraints may be due to proximity to an adjacent vehicle or other road user, and can include projected overlaps between the vehicle and other objects in the vehicle's nearby environment. Certain constraints may be binary or otherwise discontinuous in nature, in which a condition either exists at a given point in time or it does not. Noise in sensor data or prediction models may trigger such binary conditions, which in turn may cause the vehicle to alter the speed plan. Aspects of the technology employ permeable speed constraints that enable the vehicle to avoid problems associated with discontinuous constraints.