Disclosed are an optical device, an optical element, and an image display device that can achieve an improved absorption efficiency of excitation light. The optical device includes: a light-emitting element; a carrier generation layer on which light from the light-emitting element is incident and in which carriers are generated; a plasmon excitation layer that excites a plasmon, stacked on the upper side of the carrier generation layer and has a plasma frequency higher than a frequency of light generated when the carrier generation layer is excited by the light from the light-emitting element; and an exit layer that converts light or a surface plasmon generated on a surface of the plasmon excitation layer into light having a predetermined exit angle and from which the light having the predetermined exit angle exits. The optical device further includes a polarization conversion layer on the lower side of the carrier generation layer.