An electrical connector includes a housing and plural contact modules stacked adjacent to each other and held by the housing. Each contact module includes a contact array, a dielectric holder, and lossy blocks. The contact array includes signal contacts and ground contacts arranged in alternating pairs along a length of a column. The dielectric holder surrounds and engages the signal and ground contacts along intermediate segments thereof to secure the signal and ground contacts in place. The lossy blocks are mounted to the intermediate segments of the ground contacts within the dielectric holder. Each lossy block is associated with a corresponding pair of ground contacts and engages at least one of the ground contacts in the pair. The lossy blocks are composed of a lossy material that has a greater loss tangent than a low loss material that forms the dielectric holder.