A method and apparatus for processing an image based on partial images. The method includes extracting a feature of a current partial processing region of an input image frame by inputting pixel data of the current partial processing region into a convolutional neural network (CNN), updating a hidden state of a recurrent neural network (RNN) for a context between the current partial processing region and at least one previous partial processing region by inputting the extracted feature into the RNN, and generating an image processing result for the input image frame based on the updated hidden state.