A 3D ultrasonic diagnostic imaging system produces 3D display images at a 3D frame rate of display which is equal to the acquisition rate of a 3D image dataset. The volumetric region being imaged is sparsely sub-sampled by separated scanning beams. Spatial locations between the beams are filled in with interpolated values or interleaved with acquired data values from other 3D scanning intervals depending upon the existence of motion in the image field. A plurality of different beam scanning patterns are used, different ones of which have different spatial locations where beams are located and beams are omitted. In a preferred embodiment the determination of motion and the consequent decision to use interpolated or interleaved data for display is determined on a pixel-by pixel basis.