A light source device includes: excitation light sources configured to emit a plurality of excitation light including mutually different spectra; wavelength converting members configured to wavelength-convert the excitation light emitted from the excitation light sources into light having mutually different spectra and configured to be disposed in a common application region of the excitation light; and a light source control unit configured to switch a combination of the excitation light sources that are lighted among the excitation light sources based on the observation mode including a normal light observation mode and a special light observation mode to highlight a particular observation target input to an input unit, wherein light emitted from the wavelength converting members is used as illumination light, and the illumination light corresponding to the observation modes is emitted from a same emitting portion.