When a signal electron is detected by energy selection by combining and controlling retarding and boosting for observation of a deep hole, etc., the only way for focus adjustment is to use a change in magnetic field of an objective lens. However, since responsiveness of the change in magnetic field is poor, throughput reduces. A charged particle beam device includes: an electron source configured to generate a primary electron beam; an objective lens configured to focus the primary electron beam; a deflector configured to deflect the primary electron beam; a detector configured to detect a secondary electron or a reflection electron generated from a sample by irradiation of the primary electron beam; an electrode having a hole through which the primary electron beam passes; a voltage control power supply configured to apply a negative voltage to the electrode; and a retarding voltage control power supply configured to generate an electric field, which decelerates the primary electron beam, on the sample by applying the negative voltage to the sample, wherein the charged particle beam device performs focus adjustment while an offset between the voltage applied to the electrode and the voltage applied to the sample is being kept constant.