A sheet conveying device which has a reduced size and is capable of detecting the amount of lateral sheet registration deviation and correcting the deviation at high speed. The sheet conveying device includes shift roller pairs for conveying sheets, and a shift unit for shifting a conveyed sheet in a direction orthogonal to a sheet conveying direction. After the sheet has reached a position where it is to be detected by a lateral registration deviation-detecting sensor unit, a CPU causes the sensor unit and the sheet to be shifted in respective opposite directions, detect a sheet lateral edge, and thereby measure the position of the sheet lateral edge. Further, the CPU causes the sheet to be shifted based on a result of the measurement, such that the sheet lateral edge is aligned with a predetermined position.