A hearing device, e.g. a hearing aid or a headset, configured to be worn by a user at or in an ear or to be fully or partially implanted in the head at an ear of the user comprises an input unit for providing at least one electric input signal in a time-frequency representation and a signal processor comprising a neural network configured to provide respective gain values G(k,t) in said time-frequency representation for reducing noise components in said at least one electric input signal. The neural network comprises at least one layer defined as a modified gated recurrent unit, termed Peak GRU, comprising memory in the form of a hidden state vector h, and wherein an output vector o is provided by said Peak GRU in dependence of an input vector x and said hidden state vector h, wherein an output o(j,t) of the Peak GRU at a given time step t is stored as said hidden state h(j,t) and used in the calculation of the output o(j,t+1) in the next time step t+1. The signal processor is configured to provide that the number of updated channels among Nch processing channels of the Peak GRU for said input vector x(t) and said hidden state vector h(t−1) at said given time instance t is limited to a number of peak values Np, where Np is smaller than Nch. A method of operating a hearing device is further disclosed.