A system for detecting the vector of motion of a moving object on a picture screen, in which there is prepared, in addition to a picture memory for storing picture signals, a background memory for storing background picture signals. From the background memory, picture elements of a block #0b which lies at the same position on the screen as the current input block are read out and an inter-block difference B(#0', #0b) between the current input block #0' and the block #0b is calculated. If this gives rise to an uncovered background, then the above inter-block difference B(#0', #0b) between the picture memory and the background memory becomes smaller than any other inter-block differences (that is, the degree of similarity becomes the highest), and consequently, it is possible to judge that a new background portion has appeared. Incidentally, in connection with the background portion except the uncovered background, for example, a portion A' data in the picture memory and data in the background memory become identical, excepting noise components. Accordingly, in connection with this portion, the contents of the block #0 in the picture memory and the block #0b in the background memory differ little from each other, so that even if either the inter-block difference B(#0', #0) or B(#0', #0b) is chosen as a result of comparison of their magnitudes, substantially no characteristic deterioration is caused. For instance, in case of using the result of detection of movement for the movement-compensated predictive coding operation, the coding efficiency is hardly affected, even if either the block #0 or #0b is used for prediction.