In the present invention, a method for processing an image on the basis of an intra prediction mode and an apparatus therefor are disclosed. Particularly, the method for processing an image on the basis of an intra prediction mode may comprise the steps of: partitioning a block to be processed, on the basis of an intra prediction mode of the block to be processed; and performing intra prediction for the split block to be processed, wherein the direction in which the block to be processed is split is perpendicular to the prediction direction of the intra prediction mode of the block to be processed.