A vehicular trailer assist system includes a camera disposed at a vehicle and viewing at least a portion of a trailer hitched to the vehicle. A control, responsive to processing of frames of image data captured by the camera and during a calibration maneuver by the vehicle, determines an initial trailer template of trailer hitched to the vehicle. During a turning portion of the calibration maneuver, a subset of frames of image data is stored, each stored frame of image data including the trailer at a different trailer angle relative to a longitudinal axis of the vehicle. The control, responsive to storing the subset of frames of image data, converts each stored frame of image data to an edge image and matches each edge image to the determined initial trailer template, and determines a location of a hitch ball of the vehicle.