A semiconductor memory device comprises a plurality of banks, each having first and second cell mats, each having a plurality of word lines; a data access controller for selecting a word line from the first cell mat and the second cell mat in response to the row address and a refresh signal to be used in a refresh operation; and a bank controller for sequentially enabling the first cell mat and the second cell mat in response to a bank address and the refresh signal.