A retractable handle assembly includes an outer pipe, an inner pipe, an outer mounting sleeve, a control seat, a positioning post, a movable pin, a control knob, and a link. Thus, the positioning post extends through the through bore of the outer mounting sleeve and is inserted into one of the two positioning holes of the inner pipe to lock the inner pipe onto the outer pipe, so that the inner pipe and the outer pipe are combined solidly and stably to bear a greater load, thereby enhancing the strength of the retractable handle assembly. In addition, the inner pipe and the outer pipe are not in frictional contact to prevent from incurring jam between the inner pipe and the outer pipe, thereby enhancing the lifetime of the retractable handle assembly.