A vehicle configured to operate in an autonomous mode could determine a current state of the vehicle and the current state of the environment of the vehicle. The environment of the vehicle includes at least one other vehicle. A predicted behavior of the at least one other vehicle could be determined based on the current state of the vehicle and the current state of the environment of the vehicle. A confidence level could also be determined based on the predicted behavior, the current state of the vehicle, and the current state of the environment of the vehicle. In some embodiments, the confidence level may be related to the likelihood of the at least one other vehicle to perform the predicted behavior. The vehicle in the autonomous mode could be controlled based on the predicted behavior, the confidence level, and the current state of the vehicle and its environment.