Devices, systems and methods for digital video coding, which includes matrix-based intra prediction methods for video coding, are described. In a representative aspect, a method for video processing includes generating, for a conversion between a current video block of a video including multiple video blocks and a bitstream representation of the video, a most probable mode (MPM) list for a matrix based intra prediction (MIP) tool based on a rule, where, the MIP tool includes determining, during the conversion, a prediction block of the current video block by performing, on previously coded samples of the video, a boundary downsampling operation, followed by a matrix vector multiplication operation, and selectively followed by an upsampling operation, and where the rule specifies a mapping between a number of MIP modes and dimensions of the multiple video blocks; and performing the conversion based on the generating.