To remove an underline even if a business document includes a chart or even if the underline touches a character string, provided is an underline removal apparatus that removes an underline area from binary image data including the underline area touching a character string, the underline removal apparatus including: an underline search processing unit that executes a line template matching process by setting a point on the binary image data as a starting point to set a rectangular line template, tracing pixels included in the line template, and extracting a polyline indicating underline position coordinates; and an underline removal processing unit that uses the polyline to execute a process of obtaining background borderline coordinates between the underline area and a background area and character borderline coordinates between the underline area and the character string obtained by applying an interpolation process to a part in the underline area touching the character string and to execute a process of replacing an area surrounded by the background borderline coordinates and the character borderline coordinates by a color of pixels of the background area.