Background vs. foreground decisions for video frames to be compressed and transmitted in a real time video communication system are made based on a non-parametric approach using signs of pixel value changes in sequential frames. Pixel value changes are tracked as negative or positive. Cost functions may be assigned to rows and columns of predefined blocks and a decision made based on randomness of the signs within the block whether the block represents background (noise) or foreground. Recursive temporal filtering is then employed to reduce the background noise progressively resulting in increased compression and transmission efficiency. Offset tiling is used to increase accuracy of randomness determination when blocks include background and foreground combinations.