An LED lamp or bulb is disclosed that comprises a light source, a heat sink structure and an optical cavity. The optical cavity comprises a phosphor carrier having a conversions material and arranged over an opening to the cavity. The phosphor carrier comprises a thermally conductive transparent material and is thermally coupled to the heat sink structure. An LED based light source is mounted in the optical cavity remote to the phosphor carrier with light from the light source passing through the phosphor carrier. A diffuser dome is included that is mounted over the optical cavity, with light from the optical cavity passing through the diffuser dome. The properties of the diffuser, such as geometry, scattering properties of the scattering layer, surface roughness or smoothness, and spatial distribution of the scattering layer properties may be used to control various lamp properties such as color uniformity and light intensity distribution as a function of viewing angle.