A system controls a work machine that loads materials onto a conveyance vehicle. The system includes a controller. The controller performs a loading work by the work machine when the conveyance vehicle is stopped at a predetermined loading position. The controller acquires a loading amount onto the conveyance vehicle. The controller determines whether the loading work is finished based on the loading amount. The controller outputs a withdraw command to the conveyance vehicle to withdraw from the loading position upon determining that the loading work is finished. The controller determines whether the conveyance vehicle has withdrawn after outputting the withdraw command.