The invention relates to a rubber composition based on at least one diene elastomer, a reinforcing filler, a chemical crosslinker and a modifier selected from compounds comprising at least one group Q, and at least one group A, which are joined to one another by at least one and preferably one spacer group Sp, wherein: Q comprises a dipole containing at least one and preferably one nitrogen atom, A comprises an associative group comprising at least one nitrogen atom, Sp is an atom or group of atoms forming a bond between Q and A. This composition has an improved balance between stiffness under moderate deformations/elongation at break and improved hysteresis properties and is therefore especially suited to the manufacture of tires for enhancing the trade off between rolling resistance and resistance to large deformations.