A method of controlling backlight brightness, applicable to a display device including a backlight module having a plurality of dimming zones, includes: determining a target image display mode; determining a target filter core according to the target image display mode; obtaining initial backlight control data by acquiring initial backlight control values of respective dimming zones according to image data of respective image segments of a first image, wherein the respective image segments correspond to the respective dimming zones in a one-to-one correspondence; obtaining target backlight control data by performing smoothing-filtering on the initial backlight control data with the target filter core, wherein the target backlight control data comprises respective target backlight control values of the plurality of dimming zones; and controlling backlight brightness of the respective dimming zones according to the respective target backlight control values. A backlight brightness control device and a display device are further provided.