A method to derive candidate intra prediction modes, comprising: recording an intra prediction mode of at least one previous coding unit, wherein the previous coding unit is not adjacent to a to-be-processed coding unit; adding at least one recorded intra prediction mode into a set of candidate intra prediction modes, when the number of the candidate intra prediction modes is less than a preset threshold.