Disclosed is a method for processing a video signal using graph-based transform and a device for the method. A method for processing a video signal using graph-based transform, including specifying a set of available graph configurations in order to process the video signal, extracting signal structure information from samples within a processing block of the image, configuring a graph for the samples within the processing block using the signal structure information, determining transform for the processing block using the graph, and transforming the samples within the processing block using the transform. The signal structure information may be defined as information about a graph configuration which belongs to the set of specified graph configurations and which is applied to each of the samples within the processing block.