A method and device for controlling the backlight brightness of a display element are provided. The display method utilizes horizontal scan frequency and vertical scan frequency of an image to obtain a display mode of the image by referring to a scan-frequency list. When the display mode of the image is a first mode, set the backlight module of the display element to a first brightness, which makes the display element display the image with the first brightness. When the display mode of the image is a second mode, set the backlight module of the display element to a second brightness, which makes the display element display the image with the second brightness, wherein the display element displays the image more brightly with the second brightness than with the first brightness.