The invention provides a method and a device for removing rust on sheet metal. The device comprises a stander, a sheet clamping mechanism arranged on the stander and used for clamping the sheet, at least one belt grinding mechanism arranged at one side of the sheet clamping mechanism, at least one pressure mechanism arranged in the sheet clamping mechanism, a first transmission mechanism connected with the belt grinding mechanism, a second transmission mechanism of the pressure mechanism, and a control mechanism connected with the first transmission mechanism and the second transmission mechanism, wherein the control mechanism is controlled by the pressure mechanism to force a sand belt of the belt grinding mechanism to movably contact the surface of the sheet to complete the removal of ametal oxide layer on the surface of the sheet. The invention not only can solve the problem that rust cannot be removed thoroughly and manually, but also can lighten the labor intensity of workers, enhance the production efficiency, reduce the replace times of grinding material and lower the production cost.