A cutter holder includes a body, a collet, a cutter, and a fastening cap. The cutter is inserted into the collet. The collet is assembled on the body with the cutter. The fastening cap is mounted on the body, and the cutter protrudes from the fastening cap. The body has a fastening portion; the collet has an annular groove. The fastening portion has a fastening portion end face. The annular groove has a first face. The fastening portion end face and the first face are separated by a distance. By means of confining the distance between the fastening portion end face and the first face, a contact area between the body and the collet is increased to enhance the clamping force. Accordingly, the cutter holder can firmly clamp the collet for precise machining.