An inner handle assembly of a door lock includes a handle with an axle. A mounting cap has an axial hole for passage of the axle and a through-hole offset from the axial hole. A spindle driving plate has a central hole engaging fittingly a spindle of the door lock. A manually operable lock unit has a button member extending outwardly through the through-hole and mounted for movement in an axial direction between locking and unlocking positions, and an engagement member in connection with the button member. The engagement member engages the spindle driving plate for locking against rotation of the spindle driving plate when the lock unit is at the locking position. A rotary member is mounted on the axle within the mounting cap for rotating simultaneously the axle. The rotary member moves axially and outwardly the engagement member to disengage from the spindle driving plate when the axle is turned to rotate the rotary member. The axle retards rotation of the spindle driving plate so that the spindle driving plate starts to rotate after the lock unit is moved to the unlocking position.