A “snap-in” bit chuck assembly for a rotary hand or powered tool, includes a body having a hex shank at its rearward end and a retraction collar slidably disposed on the body. The body has a coaxial hex socket formed therein to allow a tool bit to be inserted thereinto. A coil spring biases the retraction collar forwardly and biases a retaining clip toward the bottom of an angular slot formed in the body. The bit is removed by sliding the retraction collar rearwardly, to compress the coil spring and allow the retaining clip to retract back up the slot. The bit is biased out of a locked position by a plunger that ensures that the bit remains disengaged when the retraction collar is released.