A technique is provided for use in a video processing system for selecting between an inter-frame coding mode and an intra-frame coding mode when processing a current portion of a video sequence. The technique includes identifying previously reconstructed samples associated with the current portion from a current frame and a reference frame of the video sequence. Then, a selection is made whether to implement the inter-frame coding technique or the intra-frame coding technique on the current portion based on comparisons associated with the previously reconstructed samples. The technique may be implemented in both a video encoder and a video decoder. As a result, the encoder does not need to transmit coding mode-related overhead information to a corresponding decoder. By way of example, the invention may be implemented in a pixel-based coding system or a block transform-based coding system.