An exterior rearview mirror assembly configured for mounting at an exterior portion of a vehicle includes a mounting arm configured for attachment at the vehicle, a mirror head disposed at an outboard end of the mounting arm and movable relative to the mounting arm, and an actuator operable to move the mirror head relative to the mounting arm. The mirror head includes a mirror casing and a frameless mirror reflective element fixedly attached at the mirror casing, with no part of the mirror casing disposed over the front surface of the mirror reflective element. The actuator is operable to move the mirror head relative to the mounting arm to vertically and horizontally adjust a rearward field of view of the driver of the vehicle who is viewing the mirror reflective element with the exterior rearview mirror assembly mounted at the vehicle.