An IC card connector assembly comprises stacked upper and lower connectors and a rear socket attached thereto. The upper connector includes an upper header and a row of upper contacts retained in the upper header. The upper header comprises a pair of upper sidebars each defining an upper recess in an outside face thereof. The lower connector includes a lower header and a row of lower contacts retained in the lower header. The lower header comprises a pair of lower sidebars each defining a lower recess in an outside face thereof. The rear socket includes an elongate base and a pair of arms upwardly extending from opposite ends of the base. The base defines two rows of cavities. In a semi-assembled state, the arms are received in the lower recesses with a gap defined therebween thereby, the rear socket being deflectably toward the lower contacts assembled to the upper and lower connectors for facilitating arrangement of the lower contacts in one of the two rows of cavities of the rear socket.