A non-planar semiconductor structure containing semiconductor fins that are isolated from an underlying bulk silicon substrate by an epitaxial semiconductor stack is provided. The epitaxial semiconductor material stack that provides the isolation includes, from bottom to top, a semiconductor punch through stop containing at least one dopant of a conductivity type which differs from the conductivity type of the particular device region that the semiconductor fin is formed in, and a semiconductor diffusion barrier layer containing no n- or p-type dopant.