A method of processing video data. The method includes down-sampling a video unit of a video prior to application of a super resolution (SR) process and performing a conversion between the video including the video unit and a bitstream of the video based on the video unit as down-sampled. A corresponding video coding apparatus and non-transitory computer-readable recording medium are also disclosed.