Methods and systems for video processing include enriching an input video feature from an input video frame set using a meta-action bank video sub-actions to generate enriched features. Reinforced image representation is performed using reinforcement learning to compare support image frames and query image frames and determine an importance of the input video frame. A classification is performed on the input video frame based on the importance and the enriched features to generate a label. An action is performed responsive to the generated label.