Disclosed is a neural network-based inference method and apparatus. The neural network-based inference method includes compressing a matrix comprising processing elements corresponding to an operation of a neural network, balancing workloads related to the operation by reordering the compressed matrix based on the workloads, and performing inference based on the reordered matrix.