An optical mouse chip with silicon retina structure comprises an image sensor array, an accumulator and a comparing/selecting unit. The image sensor array senses a direction parameter of an image along each axis. The accumulator sums the direction parameters of the image along different axes. The comparing/selecting unit selects a largest one from the sum of direction parameters of the image along different axes to determine a moving direction of the image.