A method includes placing a first plurality of device dies over a first carrier, with the first plurality of device dies and the first carrier in combination forming a first composite wafer. The first composite wafer is bonded to a second wafer, and the first plurality of device dies is bonded to a second plurality of device dies in the second wafer through hybrid bonding. The method further includes de-bonding the first carrier from the first plurality of device dies, encapsulating the first plurality of device dies in an encapsulating material, and forming an interconnect structure over the first plurality of device dies and the encapsulating material.