An exterior rearview mirror assembly for a vehicle includes a mirror head pivotally mounted at a mounting base configured for attachment at a side of a vehicle. The mirror head includes a mirror casing and a reflective element adjustably disposed at the mirror casing. A powerfold actuator is operable to pivot the mirror head relative to the mounting base between at least a folded position and a drive position. The powerfold actuator includes a primary detent that is engaged to retain the mirror head at the folded or drive position when the mirror head is pivoted via the actuator to the folded or drive position. The powerfold actuator includes a secondary detent. When the powerfold actuator pivots the mirror head between the folded and drive positions, the secondary detent is disengaged. When the mirror head is manually pivoted, the primary detent is disengaged and the secondary detent is engaged.