There are provided a high-strength hot-rolled steel sheet securing low-temperature toughness and having excellent stretch flangeability by controlling a structural fraction and a hardness difference among structures, and a manufacturing method thereof. A hot-rolled steel sheet contains: C: 0.01 to 0.2%; Si: 0.001 to 2.5% or less; Mn: 0.10 to 4.0% or less; P: 0.10% or less; S: less than 0.03%; Al: 0.001 to 2.0%; N: less than 0.01%; Ti: (0.005+48/14[N]+48/32[S]) % or more and 0.3% or less; Nb: 0 to 0.06%; Cu: 0 to 1.2%; Ni: 0 to 0.6%; Mo: 0 to 1%; V: 0 to 0.2%; Cr: 0 to 2%; Mg: 0 to 0.01%; Ca: 0 to 0.01%; REM: 0 to 0.1%; and B: 0 to 0.002%, and has: an texture in which, at a central portion of a sheet thickness located between ⅜ to ⅝ thickness positions of the sheet thickness from a surface of the steel sheet, an average value of X-ray random intensity ratios of a group of {100} to {223} orientations of a sheet plane is 6.5 or less and an X-ray random intensity ratio of a {332} crystal orientation is 5.0 or less; and a microstructure in which a total area ratio of tempered martensite, martensite and lower bainite is more than 85%, and an average crystal grain diameter is 12.0 μm or less.