Disclosed are an inter prediction mode-based image processing method and an apparatus therefor. Particularly, a method for processing an image on the basis of inter prediction may comprise the steps of: determining whether a motion vector scale adaptation is applied to a block; up-scaling a down-scaled MVD (Motion Vector Difference) when the motion vector scale adaptation is applied to the block; deriving a MV (Motion Vector) for the block, using the up-scaled MVD and a MVP (Motion Vector Predictor); and generating a predictive block of the block, using the derived MV.