A method and system for key frame based region of interest (ROI) tracking is disclosed. The method includes storing a key ROI set in a key ROI buffer, the key ROI set including at least one key ROI; designating one of the key ROI in the key ROI set as an active key ROI; receiving a point cloud representing a particular ROI to be processed for tracking; establishing a correspondence between that particular ROI and the active key ROI; determining whether to switch the active key designation to another key ROI in the key ROI set and switching the active key designation accordingly; and determining whether to modify the key ROI set and modifying the key ROI set accordingly.