A video decoding device receives a bitstream including a prediction block and a residual block with coefficients transformed by a Spatial Varying Transform (SVT). The video decoding device determines a type of SVT employed to transform the coefficients in the residual block and determines a position of the SVT relative to the residual block by determining a candidate position step size and a position index for the SVT. The video decoding device applies an inverse transform to the coefficients based on the SVT type and position to create a reconstructed residual block. The video decoding device applies the reconstructed residual block to the prediction block to reconstruct a video block and reconstructs a video sequence for display, the video sequence including a video frame that includes the reconstructed video block.