An LED lamp or bulb is disclosed that comprises a light source, a heat sink structure and an optical cavity. The optical cavity comprises a phosphor carrier having a conversion material and arranged over an opening to the cavity. The phosphor carrier comprises a thermally conductive transparent material and is thermally coupled to the heat sink structure. An LED based light source is mounted in the optical cavity remote to the phosphor carrier with light from the light source passing through the phosphor carrier. A diffuser dome is included that is mounted over the optical cavity, with light from the optical cavity passing through the diffuser dome. The diffuser dome can disperse the light passing through it into the desired emission pattern, such as omnidirection. In one embodiment, the light source can be blue emitting LED and the phosphor carrier can include a yellow phosphor, with the LED lamp or bulb emitting a white light combination of LED and phosphor light.