A method of video processing includes determining, for a conversion between a current block of a video and a bitstream representation of the video using a coding tool that accesses samples outside of the current block, that neighboring samples used for the conversion are unavailable. Samples that are located across a boundary of a video unit of the video are considered as unavailable samples. The method also includes performing, based on the determining, the conversion by padding samples in place of the neighboring samples that are unavailable. The padding samples are determined using samples that are restricted to be within a current processing unit associated with the current block.