A semiconductor device is made by providing a temporary carrier for supporting the semiconductor device. An integrated passive device (IPD) structure is formed over the temporary carrier. The IPD structure includes an inductor, resistor, and capacitor. Conductive posts are mounted to the IPD structure, and first semiconductor die is mounted to the IPD structure. A wafer molding compound is deposited over the conductive posts and the first semiconductor die. A core structure is mounted to the conductive posts over the first semiconductor die. The core structure includes a semiconductor material. Conductive through silicon vias (TSVs) are formed in the core structure. A redistribution layer (RDL) is formed over the core structure. A second semiconductor die is mounted over the semiconductor device. The second semiconductor die is electrically connected to the core structure.