Systems and methods for detecting an anomaly in an image from a set of images captured in vivo by an in-vivo imaging system may include, for each pixel of the image, associating the pixel with a color histogram value from a color histogram database; determining, for each pixel, whether the color histogram value associated with the pixel exceeds a histogram value threshold; assigning a pixel status to each pixel indicating whether the pixel is anomalous or normal; identifying one or more groups of adjacent anomalous pixels, the one or more groups of adjacent anomalous pixels each having a pixel size that exceeds a pixel size threshold; generating, using at least the one or more groups of adjacent anomalous pixels, a binary mask for the image; and determining an image anomaly score for the image based at least in part on the binary mask.