The disclosure presents a hearing device system and a method for adjusting hearing aid configuration, where a user is wearing a hearing device, the method comprising transmitting a first background sound signal to the hearing device, transmitting a first sound pattern to the hearing device, evaluating, via a camera unit, a first pupillary information of a first eye of the user in response to a first sound scene comprising the first background sound signal and the first sound pattern. The evaluating may be repeated providing at least a second pupillary information of the first eye of the user in response to a second sound scene being different from the first sound scene, and where an averaged pupillary information is provided based on an average between the first pupillary information and at least the second pupillary information, and selecting a pupillary fitting model based on the averaged pupillary information, determining a first parameter and at least a second parameter of the pupillary fitting model based on the averaged pupillary information, classifying the averaged pupillary information into either a first category or at least a second category based on the first parameter and the at least second parameter, and wherein the hearing aid configuration is adjusted if the averaged pupillary information is in the first category, and the hearing aid configuration is not adjusted if the averaged pupillary information is in the second category.