A method is provided for classifying an image. The method includes inferring location information of an object of interest in an input representation of the image. The method further includes determining foreground object features and background object features from the input representation of the image. The method additionally includes pooling the foreground object features separately from the background object features using the location information to form a new representation of the image. The new representation is different than the input representation of the image. The method also includes classifying the image based on the new representation of the image.