A system and a method of counter management and security key update for device-to-device (D2D) communication are provided. The method includes creating by a user equipment, a new packet data convergence protocol (PDCP) entity for a service group wherein a service group is identified by a destination identifier (ID), determining if any PDCP entity of the service group exists or not, generating a new proximity service (ProSe)traffic key (PTK) from a ProSe group key (PGK) corresponding to the service group associated with the new PDCP entity, initializing a new packet counter associated with the service group to zero if the new PDCP entity is a first PDCP entity associated with the service group, generating a ProSe encryption key (PEK) from the PTK and encrypting data packets mapped to the new PDCP entity using the PEK and a packet counter associated with the service group.