In accordance with a first aspect of the present invention, the sensitivity of a magnetically modulated AC-AFM is substantially improved by the use of a ferrite-core solenoid for modulating the magnetic cantilever of the ACAFM. In accordance with a second aspect of the present invention, the detection system for a magnetically modulated AC-AFM incorporates AC coupling of the signal from the position sensitive detector/beam deflection detector in order to remove the DC component of the signal, resulting in significantly improved dynamic range over systems utilizing DC coupling. High frequency modulation signals are detected through the use of fast analog multipliers which, after active filtering, give a low frequency signal which may be processed by digital electronics. In accordance with a third aspect of the present invention, operation of the microscope at small amplitudes of oscillation leaves small asperities on the tip intact and results in dramatic improvement in resolution.