A paper sheet handling apparatus includes an identification unit (20) for conveying and identifying a paper sheet A inserted from an opening (22) arranged at a front portion, a collection unit (30) for collecting the paper sheet A after identification, a pay-out unit (40) for containing a paper sheet B to be paid out to the opening (22) in a predetermined case, a convey unit (50) for conveying the paper sheet A and the pay-out paper sheet B between the identification unit (20), the collection unit (30), and the pay-out unit (40), and a frame (10) for holding the identification unit (20) located at the front side, the convey unit (50) located behind the identification unit (20), the collection unit (30) located below and in front of the identification unit (20) and the convey unit (50), and the pay-out unit (40) located behind the collection unit (30). At least the collection unit (30) and the pay-out unit (40) are detachably attached to the frame (10) from the front side.