The invention relates to a system and a method for controlling the operation of a new energy power system in an abnormal regulation domain, belonging to the technical field of renewable energy power generation and new energy grids. The invention mainly provides a control method that approximately satisfies a power difference delta P within a time delta t. The main control process comprises: averagely dividing the time delta t into three time zones delta t1, delta t2 and delta t3, and studying a control algorithm for each time zone to obtain the output level of deep regulation or energy storagewithin each period of time. The optimal control of a generator set can be quantified according to the indicator, the impact of uncertainty of new energy output on the power and frequency adjustment process of a grid can be reduced, and the role of the grid can be fully exerted to stabilize the grid.