An accessory tool retention assembly for a rotary tool includes a collet having a body defining a collet axis and a plurality of flexible segments that extend longitudinal along the body generally parallel to the collet axis. Each flexible segment includes a curved end portion that curves radially outwardly from the collet body. A collet chuck is configured to be coupled to an output shaft of a motor of a rotary tool. The collet chuck defines a collet receiving passage configured to slidably receive the collet. A detent structure is positioned to block the collet from exiting the collet receiving passage while the flexible segments of the collet are in a relaxed state.