The invention discloses a hardware acceleration method of convolution neural network inference and a device thereof. The method comprises the following steps that a control module sends residual transformation information to a path selection module; the path selection module sends characteristic map data to a dimension transformation module and/or a path buffer module; the path buffer module temporarily stores input characteristic graph data; the dimension conversion module converts the input characteristic graph data according to the number of conversion channels given by the control module;the depth convolution module accelerates the optimization of the original convolution and reduces the convolution and parameters and computational load; the path buffer module sends the temporary input characteristic map data to the residual error calculation module; the path buffer module sends the temporary input characteristic map data to the residual error calculation module; the convolution result is sent to the residual error calculation module through the dimension transformation module; the residual calculation module calculates two groups of characteristic graph data from the path buffer module and the dimension conversion module in the same dimension, thereby completing the convolution acceleration process. Compared with the prior art, the method of the invention enables the neural network to obtain a greater acceleration effect when reasoning at the mobile end.