Imaging apparatus includes a mosaic image sensor (24), which is configured to generate a stream of input pixel values belonging to a plurality of input sub-images, each sub-image responsive to light of a different, respective color that is incident on the mosaic image sensor. An image restoration circuit (26) is coupled to receive and digitally filter the input pixel values in each of the input sub-images so as to generate a corresponding plurality of enhanced output sub-images. An image signal processor (ISP) (28) is coupled to receive and combine the plurality of the output sub-images in order to generate a color video output image.