A semiconductor device is made by forming a first conductive layer over a temporary carrier. A UBM layer is formed over the temporary carrier and fixed in position relative to the first conductive layer. A conductive pillar is formed over the first conductive layer. A semiconductor die is mounted to the UBM layer to align the die relative to the conductive pillar. An encapsulant is deposited over the die and around the conductive pillar. The UBM layer prevents shifting of the semiconductor die while depositing the encapsulant. The temporary carrier is removed. A first interconnect structure is formed over a first surface of the encapsulant. A second interconnect structure is formed over a second surface of the encapsulant. The first and second interconnect structures are electrically connected through the conductive pillar. The first or second interconnect structure includes an integrated passive device electrically connected to the conductive pillar.