A holding device includes a base including a first protrusion and a second protrusion with a receiving space defined between the two protrusions. The first protrusion has a hole defined in an inside thereof and a clamping member is eccentrically received in the first protrusion so that a friction portion of the clamping member is movably extended through the hole and protrudes into the receiving space. A shank of a tool is received in the receiving space and clamped by the clamping member and a friction plate on the second protrusion. A cover is received in the first protrusion from an open rear end of the base so as to prevent foreign objects from entering the first protrusion.