The present application relates to a multi-stage encoder/decoder system that provides image compression using hierarchical auto-regressive models and saliency-based masks. The multi-stage encoder/decoder system includes a first stage and a second stage of a trained image compression network, such that the second stage, based on the image compression performed by the first stage, identify certain redundancies that can be removed from the bit string to reduce the storage and bandwidth requirements. Additionally, by using saliency-based masks, distortions in different sections of the image can be weighted differently to further improve the image compression performance.