A controller (250) for identifying out-of-plane motion of a passive ultrasound sensor (S1) relative to an imaging plane front an ultrasound imaging probe includes a memory (391) licit stores instructions and a processor (392) that executes the instructions. When executed by the processor, the instructions cause a system that includes the controller (250) to implement a process that includes obtaining (S710). from a position and orientation sensor (212) fixed to the ultrasound imaging probe (210), measurements of motion of the ultrasound imaging probe (210) between a first point in time and a second point in time. The process implemented by the controller (250) also includes obtaining (S720) intensity of signals received by the passive ultrasound sensor (S1) at the first point in time and at the second point in time based on emissions of beams from the ultrasound imaging probe (210), and determining (S730), based on the measurements of motion and the intensity of signals, directionality of and distance from the passive ultrasound sensor (S1) to the imaging plane.