The polishing apparatus has a polishing unit capable of forming a right-angled cross section by polishing a peripheral portion of the substrate. The polishing unit includes a polishing head having a pressing member configured to press a polishing tape against the peripheral portion of the substrate from above, a tape supply and recovery mechanism configured to supply the polishing tape to the polishing head and to recover the polishing tape from the polishing head, a first moving mechanism configured to move the polishing head in a radial direction of the substrate, and a second moving mechanism configured to move the tape supply and recovery mechanism in the radial direction of the substrate. The guide rollers are arranged such that the polishing tape extends parallel to a tangential direction of the substrate and a polishing surface of the polishing tape is parallel to a surface of the substrate.