Block-based motion estimation of video compression estimates the direction and magnitude of motion of objects in the scene in a computationally efficient manner and accurately predicts the optimal search direction/neighborhood location for motion vectors. A system can include a motion detection module that detects apparent motion in the scene, a motion direction and magnitude prediction module that estimates the direction and magnitude of motion of the objects detected to be in motion by the motion detection module, and a block-based motion estimation module that performs searches in reduced neighborhoods of the target block according to the estimated motion by the motion direction and magnitude prediction module and only for the blocks determined to be in motion by the motion detection module. The Invention is particularly well suited for stationary traffic cameras that monitor roads and highways for traffic law enforcement purposes.