A semiconductor device has a TSV semiconductor wafer with a cavity formed in a first surface of the wafer. A second cavity can be formed in a second surface of the wafer. A plurality of semiconductor die is mounted within the cavities. The semiconductor die can be mounted side-by-side and/or stacked within the cavity. Conductive TSV can be formed through the die. An encapsulant is deposited within the cavity over the die. A CTE of the die is similar to a CTE of the encapsulant. A first interconnect structure is formed over a first surface of the encapsulant and wafer. A second interconnect structure is formed over a second surface of the encapsulant and wafer. The first and second interconnect structure are electrically connected to the TSV wafer. A second semiconductor die can be mounted over the first interconnect structure with encapsulant deposited over the second die.