A method for performing self-interference cancellation (SIC) by an apparatus of a full duplex radio (FDR) mode in a wireless communication system including: performing a channel estimation of a received self-interference reference signal; calculating a power value of two order components of a non-linear self-interference signal based on the channel estimation; and establishing a non-linear digital self-interference cancellation order to be considered in the self-interference cancellation based on the power value of each for the two order components.