A framing member forming a frame of a factory building is provided with a slit for attaching a holder on one side face of the framing member, the slit extending in a longitudinal direction of the framing member, the slit is provided with a detachment prevention mechanism configured to exert an action of preventing an insertion piece of the holder from being detached and to fix the holder to the framing member, along with an insertion of the insertion piece into the slit, and the holder is provided with a pinching piece configured to pinch an end edge portion of a wall panel between the pinching piece and the framing member to fix the wall panel to the framing member, along with the insertion of the insertion piece into the slit.