The present invention provides a multi-channel video communication system which includes a scalable video codec, a bit-stream truncating module, a network module and a multi-channel bit-stream truncating control module. The scalable video codec encodes a video source to generate a scalable original video bit-stream. The bit-stream truncating module is set between the scalable video codec and the network module, truncates the original video bit-stream to obtain a final video bit-stream and sends the final video bit-stream to the remote client device through the network module. The multi-channel bit-stream truncating control module is connected with the bit-stream truncating module, calculates a filter parameter for each bit-stream truncating unit in the bit-stream truncating module based on received device capabilities and network conditions of the remote client device. The present invention also provides a multi-channel video communication method in the above multi-channel video communication system.