An encoder within a video transmission system controls the bit allocation at a sub-frame level. A frame is divided into smaller blocks, known as rate control blocks. Rate control blocks are used as the basic unit for bit allocation. This bit allocation achieves the target bit rate desired by the system as well as meet latency constraints. The encoder uses the slice partitioning capabilities to generate the rate control blocks using one or more slices of the image frame. This feature allows the decoder to decode the rate control blocks independently and ensures that the encoded data size for each rate control block is allocated. The encoder also detects the overflow condition for the buffer and performing an operation to avoid the overflow condition based on whether the image frame is an inter-frame or an intra-frame.