A video processing method includes performing a conversion between a video including a video unit and a coded representation of the video, where, after the video unit is encoded or decoded with an intra prediction mode, one or more frequence tables and/or one or more sorted intra prediction mode (IPM) tables are selectively updated according to a rule, where the one or more frequence tables include information about frequence of the intra prediction mode used for processing the video unit in the conversion, where the frequence indicates an occurrence of the intra prediction mode used for the conversion, and where the one or more sorted IPM tables indicate the intra prediction mode used in the processing.