The invention discloses a self-adaptive optical fiber communication nonlinear balancing method and system based on direct detection. The method comprises following steps that: known predetermined data are sent through a sender; the sent data are received at a receiver; modeling is carried out to the nonlinear balancing digital signal processing model at the receiver by using a Volterra nonlinear model according to the sent known predetermined data and the received data; the Volterra nonlinear model is equipped with N-level Volterra series; the N is a positive integer; the Volterra kernel of each level of the Volterra nonlinear model is estimated; a corresponding threshold value is set for the Volterra kernel of each level; the Volterra kernels exceeding the corresponding threshold values are eliminated from the Volterra kernel of each level so as to obtain the updated Volterra kernels; estimation is carried out to the updated Volterra kernels again so as to obtain the final Volterra kernels; and the Volterra nonlinear model is generated by using the final Volterra kernels, thus carrying out digital compensation and recovery.