This disclosure describes methods and systems for high-throughput computations in a fully-connected deep neural network. Specifically, a hardware-based deep neural network architecture including a set of parallel node processors is used to process node value transition between layers of the deep neural network, which usually involves a large-scale matrix multiplication. The set of parallel node processors are configured to decompose the large-scale matrix multiplication into sub-matrix multiplications with smaller sizes and thus reducing the hardware-complexity and making feasible direct implementation in hardware. With this implementation deep neural network may achieve a very high throughput and can handle a large number of processing layers.