The tool holder is arranged to receive a tool shank (50) of the tool (51) having a tapered surface. The holder comprises a body (10), a shank retaining collet (42) on the body and a nut (57) for causing the collet to grip the shank. The collet (42) has a shank engaging surface which, in use, lies substantially on the surface of a notional cone (50a) which tapers in the direction of withdrawal of the shank from the tool holder.