A voice over packet (VoP) system using a plurality of digital signal processors (DSPs) and a speech processing method are provided. A plurality of first-stage DSPs perform encoding voice signal of calls allocated in advance, among calls from a circuit network, and output packets corresponding to the voice signal. A plurality of first memories are disposed corresponding to the plurality of first-stage DSPs, respectively, and store feature data obtained from calls input by the plurality of first-stage DSPs, respectively. A second memory stores feature data of the number of calls exceeding the processing capacity of first-stage DSPs among the calls input from the circuit network. A second-stage DSP performs encoding voice signal stored in the second memory and outputs packets corresponding to voice signal stored in the second memory. A packet network interface collects processing results of the first-stage DSPs and the second-stage DSP and distributes inputs from a packet network to first-stage DSPs. Based on identification information added to the feature data stored in the second memory, each of the first-stage DSPs moves the feature data to the first memory disposed corresponding to the first-stage DSP.