A user terminal device, an audio system, and a method of controlling a speaker are provided. The method includes performing a pairing operation with a plurality of speakers, displaying a multi-channel setting user interface (UI) including identifiers of the plurality of speakers, and in response to a multi-channel setting command for at least two speakers from among the plurality of speakers being input through the multi-channel setting UI, transmitting a multi-channel setting command to the at least two speakers.