Provided herein are apparatus and methods for reinforcement learning based post-training sparsification. An apparatus includes: a memory; and processor circuitry coupled with the memory, wherein the processor circuitry is to: obtain a first correction parameter indicating a mean shift of a set of weights after sparsification of a model with respect to that before the sparsification of the model; obtain a second correction parameter indicating a variance shift of the set of weights after the sparsification of the model with respect to that before the sparsification of the model; and correct the set of weights at least partially based on the first correction parameter and the second correction parameter, and wherein the memory is to store the corrected set of weights. Other embodiments may also be disclosed and claimed.