In one implementation, a computer-implemented system is provided for real-time video processing. The system includes at least one memory configured to store instructions and at least one processor configured to execute the instructions to perform operations. The at least one processor is configured to receive real-time video generated by a medical image system, the real-time video including a plurality of image frames, system. The at least processor is also configured to perform an object detection to detect and obtain context information indicating an interaction of a user with the medical image at least one object in the plurality of image frames and perform a classification to generate classification information for at least one object in the plurality of image frames. Further, the at least one processor is configured to perform a video manipulation to modify the received real-time video based on at least one of the object detection and the classification. Moreover, the processor is configured to invoke at least one of the object detection, the classification, and the video manipulation based on the context information.