A PAM-N receiver capable of adaptively adjusting threshold voltages determining a level of a received signal and a method of adaptively adjusting threshold voltages thereof are disclosed. According to the method of the present invention, the result of comparison between reference data levels and the level of data in the received signal are used to adjust the reference data levels, and the threshold voltages of a PAM-N receiver are adaptively calculated from the adjusted reference data levels, thereby reflecting transmission line conditions and Inter-Symbol Interference.