A cutting tool holding device includes a base, a cutting tool holder, a screw element, a fitting positioning element and a pulling element. The base includes an accommodating groove, a receiving hole and a fitting positioning hole. The cutting tool holder includes a holder and a cutting tool receiving component. The holder includes an engaging portion detachably engaged in the accommodating groove, and a concave portion communicated with the fitting positioning hole is formed on the engaging portion. The fitting positioning element is driven by the pulling element to move in the fitting positioning hole for pressing against the concave portion tightly.