An apparatus is disclosed for removing iron powder from rolling oil used in a rolling mill has less trouble and ensures easy maintenance and long lifetime as well as improved workability. Each part combined to the apparatus can be moved and worked individually so as to facilitate repair while enhancing the effect of removing iron powder from the rolling oil. The apparatus includes a magnetic unit that is vertically supported on a frame and moved up and down by elevating means to be located inside or above the oil tank, and a scraper unit disposed at one side of the frame so as to be transverse to the magnetic unit. The scraper unit includes a scraper separating iron powder from the surface of the magnetic unit, and a chute positioned under the scraper to collect and convey iron powder.