An approach is provided for computer-vision-based object motion detection. The approach involves receiving a video sequence captured from a perspective of a vehicle/device traveling at street level. The approach also involves processing frames of the videos sequence using computer vision to determine semantic localization feature(s) associated with object(s) detected in the frames. The approach further involves clustering the object(s) into object cluster(s) across the frames based on the semantic localization feature(s). The approach further involves creating an object vector connecting the object(s) in a given cluster of the object cluster(s) across the frames. The approach further involves computing an angle between the object vector and a reference vector associated with a movement of the vehicle/device across the frames. The approach further involves classifying a motion of the object(s) relative to the vehicle/device based on the angle.