Disclosed is a fifth generation (5G) or pre-5G communication system for supporting data transmission rate higher than that of a fourth generation (4G) communication system such as long term evolution (LTE). The objective of the present disclosure is to detect a mismatch of an encryption parameter in a wireless communication system, and an operating method of a reception end comprises the steps of: receiving, from a transmission end, a packet including information related to a serial number of the packet and an encryption parameter determined on the basis of the serial number; determining whether the encryption parameter determined by the reception end is mismatched, on the basis of the information related to the serial number and the encryption parameter.