A backlight controller, a display device using the same and a method for controlling a backlight module are provided. The backlight controller includes a temporal weighting module and a dimming controlling module. The temporal weighting module calculates a weight sum of a first backlight luminance provided in a current frame and a target backlight luminance to serve as an adjusted backlight luminance. The target backlight luminance is obtained according to an image content of the current frame. The dimming controlling module increases or decreases the adjusted backlight luminance by a step value according to the target backlight luminance and generates a second backlight luminance provided in a next frame. Therefore, a backlight luminance of the backlight module not only can be smoothly adjusted to the target backlight luminance at high speed for avoiding backlight flicker, but also can be adaptive to image content for saving electric power consumption.