The present invention provides a sheet conveying device comprising a first conveyance path for conveying a sheet from a pair of first rollers to a pair of second rollers, a second conveyance path for performing switchback conveyance of the sheet, a switching unit for switching the second rollers between a press-contacted state and a separated state, and a single driving source for driving the rollers and the switching unit. A non transferring of drive section is set where when the driving source switches the rotation direction after the sheet that has been switchback-conveyed has reached the first rollers, a drive from the driving source is not transferred to the second rollers. The switching unit is driven to separate the second rollers during a time the drive is not transferred to the second rollers due to the non transferring of drive section.