The invention discloses an image restoration method based on a multi-scale structure block, and mainly solves the technical problem that distortion is easy to occur when a larger structure region is restored in the prior art. The method comprises the following realization processes: for edge pixels of regions to be restored, calculating priority through inconsistency measurement and scale invariance measurement under different scales; for the edge pixel of the region to be restored, the current priority of which is the highest, determining the size of a block to be restored adaptively according to the priority thereof; restoring the block to be restored through nonlocal reconstruction technology; and traversing the regions to be restored in the image, and repeating the steps above until finishing restoration. Structure information of image blocks in different scales is utilized, and effective structure blocks are selected for restoration, thereby improving irregular texture and random content priority judgment, and keeping image texture and structure consistency and suppressing artificial effect. A better visual effect and data index image restoration result is obtained, and the method can be used for broken image restoration and target object removal.