The technology relates to detecting and responding to processions. For instance, sensor data identifying two or more objects in an environment of a vehicle may be received. The two or more objects may be determined to be disobeying a predetermined rule in a same way. Based on the determination that the two or more objects are disobeying a predetermined rule, that the two or more objects are involved in a procession may be determined. The vehicle may then be controlled autonomously in order to respond to the procession based on the determination that the two or more objects are involved in a procession.