A high voltage semiconductor device and a manufacturing method thereof are provided in the present invention. A recess is formed in a semiconductor substrate, and a gate dielectric layer and a main gate structure are formed in the recess. Therefore, the high voltage semiconductor device formed by the manufacturing method of the present invention may include the main gate structure lower than a top surface of an isolation structure formed in the semiconductor substrate. Problems about integrated manufacturing processes of the high voltage semiconductor device and other kinds of semiconductor devices when the gate structure is relatively high because of the thicker gate dielectric layer required in the high voltage semiconductor device may be improved accordingly.