An optical semiconductor device includes a lower electrode layer formed over a semiconductor substrate, an infrared absorption layer formed over the lower electrode layer 26, and an upper electrode layer 38 formed over the infrared absorption layer 36. The infrared absorption layer includes a quantum dot having dimensions different among directions stacked, and is sensitive to infrared radiation of wavelengths different corresponding to polarization directions.