A door lock assembly includes a transverse retaining plate disposed inside a spindle tube and having a stabilizing part and a retaining tongue projecting into a retention slot in a handle. A support structure is disposed within the handle oppositely of the retention slot and has a support part extending radially into the spindle tube. When the door lock assembly is in a locking state and when the retaining tongue in the retention slot is pressed externally, one of two diametrically opposite edges of a tailpiece of a key plug abuts against the support part, and the other one of the edges abuts against the stabilizing part of the retaining plate to restrict the retaining tongue from moving away from the retention slot, thereby preventing undesirable removal of the handle from the door lock assembly.