A card edge connector (100) includes an insulative housing, a set of contacts (2) retained in the insulative housing and an ejector (3). The insulative housing has a central slot (12) and at least one tower portion (13). The tower portion includes a pair of outer walls (131) and a receiving slot (135) formed between the outer walls. Each outer wall has a ladder with a circular hole (1312) formed therein. The ejector has a pair of spindles (312) received in the circular holes and a pair of standoffs (313). Each standoff has a horizontal surface (3131) abutting against a corresponding horizontal resisting surface (1314) formed on the ladder so as to make centers of the spindle and the circular hole overlap when the ejector rotates inwardly.