A vehicular exterior rearview mirror assembly includes a powerfold actuator electrically operable to pivot a mirror head relative to a mounting base. The powerfold actuator includes a biasing element disposed between an upper housing and an upper end of a pivot tube and releasably retains the powerfold actuator in at least one detent state. The powerfold actuator includes (i) a lower detent disposed between the base portion and a lower housing, (ii) an output gear disposed between the lower housing and the upper housing and (iii) an upper detent disposed between the lower housing and the upper housing. When the powerfold actuator is in a primary detent state, the lower detent and the output gear are rotationally fixed relative to the pivot tube and the upper detent is axially movable along the pivot tube. When the mirror head is manually moved, a secondary detent state is engaged.