A hot-dip galvanized steel sheet includes a base steel sheet and a hot-dip galvanized layer formed on at least one surface of the base steel sheet, in which the hot-dip galvanized layer includes Fe in a content of more than 0% to 5% or less, Al in a content of more than 0% to 1.0% or less, and columnar grains formed by a ζ phase on the surface of the steel sheet, further, 20% or more of the entire interface between the hot-dip galvanized layer and the base steel sheet is coated with the ζ phase, and a ratio of an interface formed between ζ grains in which coarse oxides are present among grains and the base steel sheet with respect to the entire interface between the ζ phase and the base steel sheet in the hot-dip galvanized layer is 50% or less, the base steel sheet has predetermined chemical components and a refined layer in direct contact with the interface between the base steel sheet and the hot-dip galvanized layer, an average thickness of the refined layer is 0.1 to 5.0 μm, an average grain size of ferrite in the refined layer is 0.1 to 3.0 μm, one or two or more of oxides of Si and Mn are contained in the refined layer, and a maximum size of the oxide is 0.01 to 0.4 μm, and a volume fraction of residual austenite in a range of ⅛ thickness to ⅜ thickness centered at a position of ¼ thickness from the surface of the base steel sheet is 1% or more.