A system for detecting three-dimensional motion of a three-dimensional object by obtaining, for each block, a velocity vector from two temporally successive images of a moving picture including the three-dimensional object. A three-dimensional model is prepared in correspondence to the solid configuration of the three-dimensional object, an area is obtained in the previous frame picture where the three-dimensional object and the three-dimensional model overlap, the area is divided into a plurality of blocks, a velocity vector between the current input image and the previous frame image is obtained for each block, depth information in the block is obtained from the three-dimensional model, and three-dimensional motion parameters of the three-dimensional object, that is, its rotational movement and translational movement components, are obtained from the velocity vectors and depth information thus obtained in the plurality of blocks, whereby the three-dimensional motion of the three-dimensional object is detected.