A semiconductor device has a stress relief buffer mounted to a temporary substrate in locations designated for bump formation. The stress relief buffer can be a multi-layer composite material such as a first compliant layer, a silicon layer formed over the first compliant layer, and a second compliant layer formed over the silicon layer. A semiconductor die is also mounted to the temporary substrate. The stress relief buffer can be thinner than the semiconductor die. An encapsulant is deposited between the semiconductor die and stress relief buffer. The temporary substrate is removed. An interconnect structure is formed over the semiconductor die, encapsulant, and stress relief buffer. The interconnect structure is electrically connected to the semiconductor die. A stiffener layer can be formed over the stress relief buffer and encapsulant. A circuit layer containing active devices, passive devices, conductive layers, and dielectric layers can be formed within the stress relief buffer.