An object detection system and a method of detecting an object in an image are disclosed. In an embodiment, a method for detecting the object includes computing one or more feature planes of one or more types for each image pixel of the image. A plurality of cells is defined in the image, where each cell includes first through nth number of pixels, and starting locations of each cell in the image in horizontal and vertical directions are integral multiples of predefined horizontal and vertical step sizes, respectively. One or more feature plane summations of one or more types are computed for each cell. A feature vector is determined for an image portion of the image based on a set of feature plane summations, and the feature vector is compared with a corresponding object classifier to detect a presence of the corresponding object in the image portion of the image.