An accessory hub joins accessories such as towel bars and robe hooks to a wall. In order to attach the accessory to the wall, a collet is fastened to the wall. The accessory hub may be joined to the collet with a spring-loaded button extending from the collet that is inserted into a receiver in the accessory hub. A first cleat portion on the collet mates with a second cleat portion on the accessory hub further securing the accessory hub to the collet. The accessory may then be attached to the accessory hub. The accessory hub may be detached from the collet without damaging the wall by depressing the button into the collet and disengaging the first and second cleat portions.