A grinding device for raised joints of rivets on sheet metal is provided, including a lateral plate, legs, a moving plate, first springs, a first bearing seat, a sleeve, a grinding wheel, a moving bar, a 7-shaped plate, a vertical bar, a second spring, an oscillating plate, etc. The legs are fixedly connected, in a symmetric manner, to left and right sides of the bottom of the lateral plate. A through hole, which has a function of guiding, is formed on a left side of the lateral plate. The moving bar is located in the through hole. In the present disclosure, a driving motor is started to rotate the grinding wheel, the grinding device is moved to enable the grinding wheel to be located above a raised joint, and then the oscillating plate is pulled so that the grinding wheel can move downward to grind the raised joint.