A magnetic drill press includes a main housing, a base coupled to the main housing and including a magnet for selectively magnetically latching to a ferromagnetic workpiece. The magnetic drill press further includes a drill unit supported by the main housing for relative movement therewith in a direction of a rotational axis of the drill unit, a spindle assembly operable to selectively position the drill unit relative to the main housing, and a handle assembly. The handle assembly is removably coupled to the spindle assembly without the use of tools, for driving the spindle assembly.