A hitch pin retainer assembly that prevents unintended displacement of a shear pin from a trailer hitch includes a hitch plate assembly having a base plate formed with an aperture adapted to receive the shear pin assembly. The hitch plate assembly also includes a latch plate having a retainer ledge that carries a latch assembly that incorporates a receptacle projecting from the latch plate that captures a spring biased, slidable latch pin that is sized to extend, when in an extended position, above the shear pin. The latch pin is also formed with a handle at one end. The hitch pin retainer assembly can be either bolted or welded to an existing trailer hitch and also includes a tether that can be attached to the shear pin to prevent accidental loss.