A method and apparatus of sharing an on-chip buffer or cache memory for a video coding system using coding modes including Inter prediction mode or Intra Block Copy (IntraBC) mode are disclosed. At least partial pre-deblocking reconstructed video data of a current picture is stored in an on-chip buffer or cache memory. If the current block is coded using IntraBC mode, the pre-deblocking reconstructed video data of the current picture stored in the on-chip buffer or cache memory are used to derive IntraBC prediction for the current block. In some embodiments, if the current block is coded using Inter prediction mode, Inter reference video data from the previous picture stored in the on-chip buffer or cache memory are used to derive Inter prediction for the current block. In another embodiment, the motion compensation/motion estimation unit is shared by the two modes.