Systems and methods to perform behavioral planning in an autonomous vehicle from a reference state involve generating a set of actions of a fixed size and fixed order according to a predefined methodology. Each action is a semantic instruction for a next motion of the vehicle. A set of trajectories is generated from the set of actions as an instruction indicating a path and a velocity profile to generate steering angles and accelerations for implementation by the vehicle. A trajectory filter is applied to filter the set of trajectories such that unfiltered trajectories are candidate trajectories. Applying the trajectory filter includes assessing the path and velocity profile indicated by each of the set of trajectories. A selected trajectory is used to control the vehicle or the action that corresponds to the selected trajectory is used in trajectory planning to generate a final trajectory that is used to control the vehicle.