A method and apparatus for loop filter processing of reconstructed video data for a video coding system are disclosed. The system receives reconstructed video data for an image unit. The loop filter processing is applied to reconstructed pixels above a deblocking boundary of the current CTU. In order to reduce line buffer requirement and/or to reduce loop filter switching for image units, the sample adaptive offset (SAO) parameter boundary and spatial-loop-filter restricted boundary for the luma and chroma components are determined by global consideration. In one embodiment, the SAO parameter boundary and the spatial-loop-filter restricted boundary are aligned for the luma and chroma components respectively. In another embodiment, the SAO parameter boundary and the spatial-loop-filter restricted boundary for the luma and chroma components are all aligned.