An electron beam is focused on a specimen by a focusing lens. A light beam is incident on the position of irradiation of the specimen with the electron beam and the reflected light beam is detected by a linear light detector. The output of the detector is used to measure the height of the specimen at the position of irradiation of the electron beam. The specimen is moved in a plane perpendicular to an optical axis of the focusing lens. A specimen height measuring device carries out the height measurement of the specimen at a position to be observed on the specimen and at positions thereon which are in the vicinity of the position to be observed, when those positions are located at the position of irradiation of the electron beam. The specimen height measuring device averages the measured values so as to produce a focusing correction signal on the basis thereof and controls the focusing lens on the basis of the focusing correction signal.