An encoding manager facilitates robust single-pass variable bit rate video encoding of a video sequence. Before encoding the video sequence, the encoding manager determines the size of a buffer to use for keeping track of over/under used bits generated during encoding. The encoding manager uses the target bit rate for the video sequence and the length of the video sequence to determine the size of the buffer. After allocating bits to a frame of the sequence, the encoding manager determines the quant to use to encode that frame. The determination of a quant to use to encode a frame is informed by the fullness of the buffer. The encoding manager adjusts the quant to use (and thus the aggressiveness of its encoding) in response to the amount of overflow generated thus far by the encoding of the video sequence.