An exhaust purification system of an internal combustion engine which has a plurality of cylinders is comprised of an exhaust purification catalyst, a downstream side air-fuel ratio sensor, and a control device which controls the average air-fuel ratio of the exhaust gas and the combustion air-fuel ratios of the cylinders. The control device performs average air-fuel ratio control where it alternately controls the average air-fuel ratio between the rich air-fuel ratio and the lean air-fuel ratio and inter-cylinder air-fuel ratio control where it controls the combustion air-fuel ratios of the cylinders so that the combustion air-fuel ratio becomes the rich air-fuel ratio at least at one cylinder among the plurality of cylinders even when the average air-fuel ratio is controlled to the lean air-fuel ratio by average air-fuel ratio control. In average air-fuel ratio control, the average air-fuel ratio is controlled so that the lean shift amount when controlling the average air-fuel ratio to the lean air-fuel ratio becomes smaller than the rich shift amount when controlling the average air-fuel ratio to the rich air-fuel ratio.