A system and method for classification of images of an image stream may include receiving an image stream of unclassified images, for example produced by an in-vivo imaging device, and based on indirect user input, adapting an initial classification algorithm to classify images to groups based on at least a subset of the received image stream of unclassified images. The indirect user input may be used to generate user-based indications for the classification.