A method and system of extracting a perceptual feature set for image/video segmentation are disclosed. An input image is converted to obtain a hue component and a saturation component, where the hue component is quantized into a number of quantum values. After weighting the quantized hue component with the saturation component, the weighted quantized hue component and the saturation component are subjected to a statistical operation in order to extract feature vectors. Accordingly, the method and system provide overall segmentation results that are very close to human interpretation.