The invention discloses a brightness adjusting device and method of a display device. The device comprises an environment information sensing unit, an environment information extracting unit, a central processing unit and a driving unit, wherein the environment information sensing unit is used for obtaining an image in front of the display device and transmitting the image to the environment information extracting unit; the environment information extracting unit is used for analyzing and extracting brightness information of an environment and a figure existing condition, and transmitting the brightness information of the environment and the figure existing condition to the central processing unit; the central processing unit is used for generating brightness adjusting data according to the brightness information of the environment and the figure existing condition and transmitting the brightness adjusting data to the driving device; the driving device is used for adjusting the brightness of a backlight source according to the brightness adjusting data. According to the brightness adjusting device, the environment information in front of the display device is sensed; when the environment light is stronger, the brightness of the display device is increased; when the environment light is darker, the brightness of the display device is reduced so that the watching comfort of a user is improved; when no people watch, the backlight is turned off or reduced to a pre-set value; the backlight is immediately turn or the brightness is recovered when people come to watch, so that the use power consumption of the display device is reduced.