The invention discloses a rapid image three-dimensional matching method and device. The method includes the following steps that a reference view is over-segmented into a plurality of areas which are not overlapped with one another; the pixel initial matching cost between a pixel i in the reference view and a pixel i' in a target view is calculated; a candidate parallax subset and a support subset at the position of the current pixel i are determined; iterative computation is conducted on the aggregation matching cost of the pixel i, the candidate parallax subset and the support subset are updated and reduced according to the aggregation matching cost, and elements in the candidate parallax subset and elements in the support subset are reduced continuously until the number of candidate parallax in the candidate parallax subset is reduced to a set parallax number; the parallax value corresponding to the maximum matching cost in the candidate parallax subset is selected as a first parallax value of the pixel i. According to the rapid image three-dimensional matching method and device, the matching accuracy is guaranteed, and meanwhile, the algorithm running speed is greatly increased; in addition, a parallax plane fitting method based on the PatchMatch algorithm is used, and therefore the optimal plane parameter estimation value can be converged rapidly.