What is disclosed is a system and method for determining an orientation direction of a color edge at a given pixel location in a binary color image. The orientation direction of the color edge is determined from eight pixel counts with each pixel count being a total number of pixels in each of eight regions of a window centered about a candidate pixel which resides along the color edge. The eight regions are associated with 8 compass points. The orientation of the edge is determined by a 1st, 2nd and 3rd tier control bits which are based upon the pixel counts of each region. The 1st, 2nd and 3rd tier control bits collectively form a 3-bit word. The 3-bit word defines the orientation direction. The teachings hereof provide an efficient way of performing binary edge orientation detection by making uses of intermediate results to simultaneously encode the edge orientation.