The disclosure relates to fifth generation (5G) or sixth generation (6G) communication systems to support higher data rates. A receiver of a communication system includes a transceiver configured to receive a signal encoded based on polarization-adjusted convolutional (PAC) coding from a transmitter, and a controller configured to calculate branch metrics of branches defined in a trellis associated with the PAC, calculate a path metric based on an accumulated sum of the branch metrics, determine a path assigned to each state based on the path metric, and estimate a codeword based on the determined path, wherein a number of paths assigned to each state is dynamically determined according to the number of active states, and wherein the active state includes at least one branch.