A signal processing apparatus which processes a voice packet is provided. The signal processing apparatus includes: a network jitter measurer which estimates network jitter using a weighting of a network jitter variance according to a network state, a compression and output unit which compresses or outputs a voice frame based on the estimated network jitter, and a loss concealer which divides a voice packet loss into short-term loss concealment and long-term loss concealment, and performs loss concealment in a different way depending on whether the voice packet loss is the short-term loss concealment or the long-term loss concealment.