Disclosed are an intra-prediction mode-based image processing method and a device therefor. Particularly, a method for processing an image on the basis of an intra-prediction mode may comprise the steps of: configuring a reference sample to be used for prediction of a current block on the basis of width information and height information of the current block when the current block is a non-square block; deriving an intra-prediction mode of the current block; and generating a prediction sample of the current block by using the reference sample on the basis of the intra-prediction mode of the current block.