An assembly of saw handle and saw member is provided. An end portion of the saw member is inserted into an inserting groove of the saw handle. The saw handle has stopping bars at an inserting portion and a swing arm. The end portion defines an opened hole and a closed hole. When the swing arm is moved to an eject position and not received in a receiving space of a guard portion of the saw handle, the saw member is not blocked by the swing arm and is allowed to be inserted into the inserting groove. After the saw member is inserted into the inserting groove and the swing arm is moved to a fix position, the stopping bars are respectively located in the opened hole and the closed hole, so that the saw member can be assembled with the saw handle in a convenient and rapid manner.