A semiconductor device has a thermally conductive layer with a plurality of openings formed over a temporary carrier. The thermally conductive layer includes electrically non-conductive material. A semiconductor die has a plurality of bumps formed over contact pads on the die. The semiconductor die is mounted over the thermally conductive layer so that the bumps are disposed at least partially within the openings in the thermally conductive layer. An encapsulant is deposited over the die and thermally conductive layer. The temporary carrier is removed to expose the bumps. A first interconnect structure is formed over the encapsulant, semiconductor die, and bumps. The bumps are electrically connected to the first interconnect structure. A heat sink or shielding layer can be formed over the semiconductor die. A second interconnect structure can be formed over the encapsulant and electrically connected to the first interconnect structure through conductive vias formed in the encapsulant.