The invention provides a technique for eliminating vertical parallax of multi-viewpoint stereo image. The technique comprises the steps of putting a reference flat plate with a rectangle pattern in a photographing scene, using a camera array to photograph the reference flat plate and calibrating four vertexes of the rectangle, calculating the coordinate mapping relationship of the same object point corresponding to different image photographing points one point by one point according to the four vertexes of the rectangle photographed by each camera, then moving the reference flat plate away and photographing other scenes by the camera array, rearranging the pixel of the image photographed by each camera according to the coordinate mapping relationship, so as to obtain the image of each viewpoint with vertical parallax eliminated. The technique is also used for multi-viewpoint free stereo display.