Encoding and decoding using prediction dependent transform coding are provided. Encoding and decoding using prediction dependent transform coding may include identifying a current input block from a current input frame from an input video stream, generating a prediction block for the current input block, generating a residual block based on a difference between the current input block and the prediction block, generating, by a processor in response to instructions stored on a non-transitory computer readable medium, an encoded block by encoding the residual block based on the prediction block using the using prediction dependent transform coding, including the encoded block in an output bitstream, and outputting or storing the output bitstream.