Method and apparatus of managing a DPB (decoded picture buffer) for a video coding system are disclosed. Whether the IntraBC mode is used for a current picture is determined and a maximum allowed number of decoded pictures in the DPB is allocated based on whether the IntraBC mode is used for the current picture. The current picture is then encoded or decoded using decoded pictures stored in the DPB as reference data. In another method, a maximum number of reordered pictures in the DPB is determined. If certain conditions are true, one or more decoded pictures stored in the DPB will be emptied until none of the conditions is true. One condition corresponds to a number of decoded pictures in the DPB marked as “needed for output” being greater than the maximum number of reordered picture modified according to whether the IntraBC mode is used for the current picture.