The invention provides a high-resolution remote sensing image multi-scale segmentation optimization method based on spectral difference maximization, and the method comprises the following steps: setting a plurality of different segmentation scales for a given high-resolution remote sensing image, and obtaining initial segmentation results under different scales; establishing a segmentation objectadjacent graph RAG; calculating the local spectral difference of each segmentation object by using the spectral angle, including the spectral difference between the segmentation object and the interior of the segmentation object, and obtaining the spectral angle difference value of each segmentation object; generating corresponding spectral difference value curves of different segmentation scalesat the same position, and selecting a corresponding segmentation object at the scale with the highest spectral difference value as an optimal segmentation object at the position; and combining the segmentation objects selected at all positions to form a final segmentation optimization result. According to the method, the optimal segmentation object is directly selected from the initial segmentation results under multiple segmentation scales, the optimal segmentation scales of the remote sensing image under different ground features are adaptively adjusted, and an ideal segmentation effect isobtained.