A method and apparatus of line buffer reduction for context adaptive entropy processing are disclosed. The context formation for context adaptive entropy processing depends on block information associated with one or more neighboring blocks. When the neighboring block is on a different side of a region boundary from the current block, the block information is replaced by replacement block information to reduce or remove line buffer requirement for storing the block information of neighboring blocks on the other side of the region boundaries from the current block. The context adaptive entropy processing is CABAC encoding, CABAC decoding, CAVLC encoding, or CAVLC decoding.