A sheet processing apparatus includes a conveying roller that conveys a sheet, a processing tray on which the sheet from the conveying roller is loaded, a reference stopper provided at one end of the processing tray, a return paddle having an elastic piece for transferring the sheet from the conveying roller to the reference member, and a roller arm that moves the return paddle in the sheet thickness direction at a predetermined moving rate according to the number of sheets loaded on the processing tray. The moving rate of the roller arm is reduced as the number of sheets loaded on the processing tray is increased. With this configuration, aligning property of even a wavy sheet can be suppressed from being deteriorated at sheet loading.