Disclosed herein are an inter prediction mode-based image processing method and an apparatus therefor. Specifically, a method for processing an image based on an inter prediction mode may include: generating a temporary candidate list by using a spatial neighbor block or temporal neighbor block of a current block; calculating a cost for each candidate in the temporary candidate list by using a template of the current block; generating a final candidate list by rearranging candidates of the temporary candidate list based on the calculated cost; decoding a merge index indicating a specific candidate in the final candidate list; and generating a prediction block of the current block by using motion information of a candidate indicated by the merge index.