An accessory mounting system for a vehicle includes a mirror mounting button attached at an in-cabin surface of the vehicle windshield and a mounting element attached at the in-cabin surface of the windshield separate from the mounting button. An interior rearview mirror assembly has a mirror head and a mirror support, which includes a mirror mount configured to mount the mirror assembly to the mirror mounting button. A module is configured to be received at and supported by the mounting element attached at the windshield. The module accommodates a forward facing camera that views through the windshield. The forward facing camera may be part of a driver assistance system of the vehicle. With the module received by and supported by the mounting element, the mirror mount can be mounted to the mirror mounting button and can be demounted from the mirror mounting button without demounting the module from the mounting element.