A semiconductor device includes a pair of sensor units each of which includes a photoelectric conversion unit, a signal holding unit and a transfer unit, and outputs a signal held by the signal holding unit, comprising a control unit including a detector unit, wherein when one of the pair of sensor units operates in a first mode, the other operates in a second mode, the detector unit detects that the output has reached a predetermined value after the one starting a signal transfer, the one ends the signal transfer in response to the detection and determines the held signal, the control unit generates a control signal after that, and the other in the second mode accumulates generated charges and starts a signal transfer in accordance with the control signal, then ends the signal transfer and determines the held signal.