A control processor for a video encode-decode engine is provided that includes an instruction pipeline. The instruction pipeline includes an instruction fetch stage coupled to an instruction memory to fetch instructions, an instruction decoding stage coupled to the instruction fetch stage to receive the fetched instructions, and an execution stage coupled to the instruction decoding stage to receive and execute decoded instructions. The instruction decoding stage and the instruction execution stage are configured to decode and execute a set of instructions in an instruction set of the control processor that are designed specifically for accelerating video sequence encoding and encoded video bit stream decoding.