After initial intra prediction based on projection along an intra prediction direction, there may exist discontinuities between the predicted block and neighboring blocks. To smooth the block boundary, boundary filtering may be used for intra prediction. In particular, different methods that use multiple reference layers may be used for boundary filtering. Which filtering method to be used may be selected adaptively based on the intra prediction mode. When gradient-based filtering is selected, the gradient at a reference sample, that is at the same row or column as the current sample, in a reference layer is calculated as the difference between the reference sample and the predicted value for the reference sample using the intra prediction mode. Additional gradient at another reference sample from another reference layer may also be used. Then the initially predicted value for the sample maybe adjusted based on the gradient(s) and a decay function.