In one embodiment, a method receives a video bitstream corresponding to compressed video, wherein Filter Unit (FU) based in-loop filtering is allowed in a reconstruction loop associated with the compressed video. The method then derives reconstructed video from the video bitstream, wherein the reconstructed video is partitioned into FUs and derives a merge flag from the video bitstream for each of the FUs, wherein the merge flag indicates whether said each of the FUs is merged with a neighboring FU. The method further receives a merge index from the video bitstream if the merge flag indicates that said each of the FUs is merged, and receives the filter parameters from the video bitstream if the merge flag indicates that said each of the FUs is not merged. Finally, the method applies the in-loop filtering to said each of the FUs using the filter parameters.