The invention relates to a method of 3D reconstruction of a scene by means of 2D panoramic images of the scene, which comprises a step of processing these images. These images arise from a panoramic system moving in displacement along a determined trajectory, such that the image of at least one point of the scene is in at least 3 successive images obtained according to various directions; the step of processing these 2D successive images comprises the sub-steps: a) determining reconstruction planes in the scene to be reconstructed, b) determining, on the basis of pairs of panoramic images and for each pair, rectification planes corresponding to the reconstruction planes and projecting onto each of them a sector of each image of the pair, in a direct manner, so as to obtain two 2D rectified images, c) matching the two 2D rectified images so as to obtain an intermediate 3D reconstruction, d) transforming each intermediate 3D reconstruction into a 3D frame including the reconstruction planes so as to obtain a transformed intermediate 3D reconstruction, e) repeating steps b) to d) on the basis of a new pair of 2D panoramic images and of at least one other rectification plane, so as to obtain at least one other transformed intermediate 3D reconstruction, f) temporally fusing at least two transformed intermediate 3D reconstructions so as to obtain a 3D reconstruction of the scene.