A hot rolled steel sheet has a chemical composition including, by mass %, C: 0.060% to 0.120%; Si: 0.10% to 0.70%; Mn: 1.00% to 1.80%; P: 0.10% or less; S: 0.010% or less; Al: 0.01% to 0.10%; N: 0.010% or less; Nb: 0.010% to 0.100%, wherein Nb is contained so that content of solute Nb is 5% or more relative to the total Nb content; the balance being Fe and incidental impurities. The hot rolled steel sheet has a microstructure containing ferrite of not more than 15 μm in average crystal grain diameter by a volume fraction of not less than 75%, the balance being low-temperature-induced phases. The hot rolled steel sheet can be suitably utilized for manufacturing a cold rolled steel sheet or hot-dip galvanized steel sheet having a tensile strength of 590 MPa or more, excellent in material homogeneity and capable of giving excellent cold rolling property.