A manual sheet feeder that is an example of a sheet conveying device includes a roller unit in which a rubber roller is mounted on a support member. A separation wall is formed in an internal space of the support member, and a rotation shaft is formed at the separation wall. A slide shaft is provided such that a coil spring is interposed between the rotation shaft and the slide shaft in a compressed state. Movement of the slide shaft from an opening outward in an axial direction is restricted by an elastic engagement piece of a movement restriction mechanism. The elastic engagement piece is formed at a spring holding portion of the slide shaft. In a state where the coil spring is stored in the spring holding portion, a back surface of the elastic engagement piece is supported by the coil spring.