The invention discloses multi-channel video stream encoder and decoder based on the depth image rendering. When encoding the video stream, the multi-channel video stream encoder and decoder can also encode the central channel video stream image frame and the depth map according to the video encoding standard method; the predicting map of the neighboring auxiliary channel to be encoded can be obtained by adopting the depth image rendering technology according to the image frame and the depth map after the channel is reconstructed, and converting, quantizing and entropy encoding can be performed to the shielding information. When the multi-channel video compression code is decoded by the multi-channel video stream encoder and decoder, the physiologic tendency of the human visual system can be utilized, and a virtual channel positioned at the center position of the two neighboring channels can be obtained by adopting the depth image rendering technology according to the image frames and the depth maps of the two neighboring channels. At the display terminal where the decoder of the invention is adopted to output the video stream, each vertical visual point is composed of a high-quality channel and a virtual channel, and the audience can get the good vertical visual experience.