A method for controlling a hearing device via touch gestures carried out by a wearer of the hearing device. As part of the gesture a finger is swiped across first and second sound inlets of the hearing device. A gesture is detected by determining if a first output signal from a first microphone at the first sound inlet contains a component caused by the finger touching the hearing device at the first sound inlet and if a second output signal from a second microphone at the second sound inlet contains a component caused by the finger touching the hearing device at the second sound inlet. Based on the detected gesture a control signal changes a processing parameter or setting of a signal processing unit of the hearing device. Additionally, a hearing device capable of performing the proposed method and a method of fitting such a hearing device are provided.