Disclosed herein are an intra prediction mode based image processing method and an apparatus therefor. Specifically, a method for processing an image based on an intra prediction mode may include: deriving an intra prediction mode of a current block; generating a bottom right reference sample adjacent to a bottom right side of the current block; generating a right reference sample or a lower reference sample by using the bottom right reference sample generating a first prediction sample and a second prediction sample of the current sample in the current block based on a prediction direction of the intra prediction mode; and generating a final prediction sample of the current sample by interpolating the first prediction sample and the second prediction sample.