A semiconductor memory device includes a memory cell array having plural memory cells that require a refresh operation when retaining data; a read/write control unit that performs read-access or write-access of memory cell address specified for the memory cell array based on instructions from the outside; a refresh control unit that performs hidden-refresh of memory cells without control from the outside; and a schedule control unit that makes the refresh control unit perform hidden-refresh after the read/write control unit read-accesses the memory cell array, and that also makes the refresh control unit perform hidden-refresh before the read/write access control unit performs write-access.