A hydrogenation catalyst system is provided. The catalyst system includes a metal complex of Formula (I), an organic lithium compound and an organic compound having a cyclic structure including at least one double bond. In Formula (I), M is transition metals. R1, R2, R3, R4 and R5 are the same or different, including hydrogen, C1-8 alkyl, and C1-8 alkoxy, or two of R1, R2, R3, R4 and R5 are linked together to form a ring. X1, X2 and X3 are a cyclic group, hydrogen, chlorine, bromine, alkyl or alkoxy, wherein when one of X1, X2 and X3 is a cyclic group, and the others are the same or different, including hydrogen, chlorine, bromine, alkyl or alkoxy. The invention also provides a selective hydrogenation process utilizing the catalyst system.