System and method for automatically rating the content of video media based on video operations performed on a media device and in reference to a plurality of rating rules are provided. Usage of the media device is continuously monitored and user actions with respect to operating the video media on the media device are automatically logged. Each rating rule includes a device usage pattern with respect to operating videos on the media device and a rating action indicating adjustments to content ratings of the videos based upon characteristics described by the device usage pattern that are inferred from the recorded user inputted video control operations. When the device usage pattern of a rating rule is inferred from one or more user actions operating a piece of video media directly on the media device, the content rating of the piece of video media is adjusted based on the rating rule.