The present disclosure relates to a system for controlling vibration of an engine including an engine inertia portion which rotates together with the engine and a sub-inertia portion which influences a rotation speed of the engine and is separately provided, and a damper which is disposed between the engine inertia portion and the sub-inertia portion for reducing a vibration. The present disclosure includes: determining whether a vehicle is in an idle state; sensing rotation speeds of the engine inertia portion and the sub-inertia portion, respectively; calculating an average value of the rotation speeds of the engine inertia portion and the sub-inertia portion; calculating an error value from the average value; and PI controlling by receiving the error value.