Provided is a segmentation and tracking system based on self-learning using video patterns in video. The present invention includes a pattern-based labeling processing unit configured to extract a pattern from a learning image and then perform labeling in each pattern unit to generate a self-learning label in the pattern unit, a self-learning-based segmentation/tracking network processing unit configured to receive two adjacent frames extracted from the learning image and estimate pattern classes in the two frames selected from the learning image, a pattern class estimation unit configured to estimate a current labeling frame through a previous labeling frame extracted from the image labeled by the pattern-based labeling processing unit and a weighted sum of the estimated pattern classes of a previous frame of the learning image, and a loss calculation unit configured to calculate a loss between a current frame and the current labeling frame by comparing the current labeling frame with the current labeling frame estimated by the pattern class estimation unit.