A tracking system and method for evaluating whether image information for a region cluster of a video frame of a scene represents an hypothesis of an object to be tracked, such as a person. At least one real-world feature of a region cluster corresponding to an object to be tracked is generated. For example, the at least feature is at least one possible location of a predetermined portion of an object represented by the region cluster is determined based on a viewing angle of the scene of the video camera. A distance from the video camera to the object corresponding to the region is determined in real-world coordinates for each possible location of the predetermined portion of the region cluster. Real-world size and location information for the region cluster is determined based on the distance. The real-world size and location information for the region cluster is compared with statistical information for the particular type of object to determine a confidence value representing a measure of confidence that the region cluster represents the particular type of object.