The virtual model control system has an input device configured to provide input information for formation, movement or transformation of a virtual model; a control device configured to form the virtual model based on the input information received from the input device, move or transform the virtual model, form a plurality of physical particles at the virtual model, form contact point information therefor, and move the plurality of physical particles to update the contact point information; an output device configured to output the virtual model to the outside. When the plurality of physical particles penetrates into another virtual model in the virtual space, the control devices update the contact point information so that the penetrating physical particles are rearranged at an outer side of the another virtual model.