In a handle-returning device of a cylinder lock assembly, a support base has an inner annular flange and an axial spring-retaining lug. A rotation driver has an inner tubular wall extending into the inner annular flange of the support base, and a spring driving element. A torsion spring is disposed around the inner annular flange and between the support base and the rotation driver, and has a spring leg abutting the spring-retaining lug and the spring driving element. A rotary plate abuts the support base opposite to the rotation driver, and has an engaging part engaging a tongue projecting from the rotation driver. A reinforced structure is thus provided to oppose high torsional stresses produced upon rotation of a lever handle.