A stroke-guided vectorization system is described that generates, from an input sketch and guide image depicting an approximate vector representation of the sketch, an aligned guide image depicting an improved vector representation of the sketch. To do so, the stroke-guided vectorization system determines black levels representing a vector stroke in the input sketch and white levels representing a background in the input sketch. The stroke-guided vectorization system determines a black threshold value and a white threshold value for discrete portions of the aligned guide image using subsets of the black levels and subsets of the white levels determined using the input sketch. Each discrete portion of the aligned guide image is then mapped to a vector stroke or a background based on the black threshold value and the white threshold value of the portion.