A control apparatus comprises a housing and is adapted to control a hearing device by recognizing predefined gestures made by the device wearer by moving one arm and/or or hand relative to the housing when the housing is in an operating position at or on the wearer's body. The housing comprises a reference electrode coupled capacitively to the wearer when the housing is in the operating position and a first sensor electrode. The control apparatus further comprises: a first signal generator to provide a first electric probe signal between the first sensor electrode and the reference electrode; a first measurement circuit to determine first signal values in dependence on the impedance between the first sensor electrode and the reference electrode; a detector to recognize gestures in dependence on the first signal values; and a control unit to provide control commands to the hearing device in dependence on recognized gestures.