A method of video processing includes deriving, for a conversion between a chroma block of a video and a bitstream representation of the video, parameters of a cross-component linear model by using downsampled luma samples that are generated from N above neighboring lines of a collocated luma block of the chroma block using a downsampling filter, where N is a positive integer; and performing the conversion using a predicted chroma block generated using the cross-component linear model.