Disclosed is a roller hemming system for panels of a vehicle that may include: a marriage jig adapted to load thereon a part having an outer panel and an inner panel joined together,a clamping pad releasably mounted at a front end of an arm of a first handling robot, and adapted to clamp the part loaded on the marriage jig by a plurality of clamping means,a plurality of transfer rails adapted to intersect with each other at a center,a plurality of hemming dies, each being disposed on a predetermined transfer rail, being adapted to reciprocate on the transfer rail between the center and an initial position, and being adapted to receive thereon the part clamped by the clamping pad,and a plurality of hemming robots being disposed corresponding to each hemming die and provided with a hemming roller unit adapted to hem an edge of the part.