A method and system for constructing a convolutional neural network (CNN) model are herein disclosed. The method includes regularizing spatial domain weights, providing quantization of the spatial domain weights, pruning small or zero weights in a spatial domain, fine-tuning a quantization codebook, compressing a quantization output from the quantization codebook, and decompressing the spatial domain weights and using either sparse spatial domain convolution and sparse Winograd convolution after pruning Winograd-domain weights.