The process is characterized in that it comprises the following steps: segmentation of the sequence into alternating video shots, classification of these shots according to camera angles in order to obtain classes, construction of a sprite or video object plane for a class that is an image corresponding to the background relating to this class, grouping of at least two sprites onto the same sprite or video object plane, in order to form an image called large sprite, extraction, for the shots corresponding to the large sprite, of image foreground objects from the sequence relating to these shots, separate encoding of the large sprite and of the extracted foreground objects. Application to the transmission and to the storage of video data.