A video hardware engine with multi-threading functionality is disclosed. The video hardware engine includes a video hardware accelerator unit and a controller. The controller is coupled to the video hardware accelerator unit. The controller operates in an encode mode and a decode mode. In the encode mode, the controller receives a plurality of frames and encode attributes associated with each frame of the plurality of frames. The encode attributes associated with a frame of the plurality of frames is processed to generate encode parameters associated with the frame. The video hardware accelerator unit is configured to process the frame based on the encode parameters to generate an output. The output of the video hardware accelerator unit is processed to generate a compressed bit-stream and an encode status. In decode mode, the controller receives a compressed bit-stream and decode attributes and generates a plurality of frames and a decode status.