Disclosed are a highlight image restoration method and device based on an optical field. The method comprises a depth estimation step, a highlight detection step and a highlight removing step; in the depth estimation step, optical-field image depth estimation is performed through combination of defocusing and stereo viewpoint matching; in the highlight detection step, highlight detection is performed based on diagonal multi-viewpoint color space projection differences; and, in the highlight removing step, depth values corresponding to detected highlight points are obtained from a depth estimation result, macro pixels corresponding to the highlight points are found through refocusing, pixels in the macro pixels cluster into two types according to brightness, equation sets are built for the two types by utilizing double-color reflection models, specular reflection components of the highlight points are obtained after the equation sets are solved, the specular reflection components are removed, and then highlight removing is achieved. The device comprises a depth estimation module, a highlight detection module and a highlight removing module which achieve the steps. The device can accurately identify the highlight pixels, and a recovery result has the advantages of good authenticity and robustness.