The present application provides an optical semiconductor device with a composite intervening structure. The optical semiconductor device includes a logic die including a core circuit area and a logic peripheral circuit area; a memory die positioned on the logic die and including a memory cell area and a memory peripheral area, and a first inter-die via positioned in the memory peripheral area and electrically connected to the logic peripheral circuit area; and a sensor die positioned on the memory die and including a sensor pixel area and a sensor peripheral area, a first intra-die via positioned in the sensor peripheral area and electrically coupled to the logic peripheral circuit area through the first inter-die via, and a second intra-die via positioned in the sensor peripheral area. The intervening structure is disposed on the back surface of the memory die.