A tool holder for releasably coupling a bit with a non-circular outer connector surface to a rotary driver. The tool holder includes a socket and a retaining mechanism coupled to the socket and operable in a retaining mode and a released mode. The socket includes a forward end with an opening extending rearwardly from the forward end. The opening also includes a non-circular cross-section adapted to receive the outer connector surface of the bit to couple the bit for rotation with the socket. The retaining mechanism is adapted to permit disposable of the bit into an operative position within the opening and to prevent axial movement of the bit out of its operative position when the retaining mechanism is in its retaining mode. The retaining mechanism is further adapted to permit axial displacement of the bit out of its operative position when the retaining mechanism is in its released mode. The retaining mechanism includes a sleeve coupled to the socket for displacement toward the forward end of the socket from a locked position to an unlocked position thereby moving the retaining mechanism from its retaining mode into its released mode. The retaining mechanism also includes a spring or alternately a rubber o-ring biasing the sleeve into its locked position.