A ball-lock tool retainer insert includes a retainer insert body having a generally flat top and bottom face arid a generally elongated non-circular cross-section. A lower portion and an upper portion of the retainer insert body each have a constant cross-section with the upper portion cross-section being larger than the lower portion crosssection thereby forming an enlarged shoulder. The retainer insert body has a tool aperture with a constant cross-section extending from the top face to the bottom face for receiving a tool and an angle hole adjacent the tool aperture extending from the top face and partially extending into the tool aperture for holding a ball lock. The retainer insert body is adapted to be received by a retainer block having a generally elongated non- circular hole with a counterbore such that the enlarged shoulder of the retainer insert body fits into the counterbore.