A connector assembly for connecting to differential signal circuits on at least one circuit board may be of the docking type or the interposer type. A housing of the connector has transversely and longitudinally extending walls that define a plurality of cavities that extend completely through the housing. Terminal assemblies with at least one differential signal pair are received in the cavities with tail portions on at least one end of the differential pairs to connect to a circuit board. The connector housing has conductive surfaces and a plurality of ground terminal members are disposed along the transverse walls or the longitudinal walls, or both, for connecting to ground circuits in the circuit board. The ground terminal members each have a plurality of ground terminals, which are disposed along the walls at intermediate or diagonal positions relative to the differential signal pairs, such as adjacent to four differential signal pairs. Such placement of the ground terminal members also subdivides the differential signal pairs into groups or sets. This interstitial ground terminal arrangement also serves to keep the impedance at the connector-circuit board interface relatively uniform for high speed signals in the differential pairs.