Described is a technology by which phishing-related data sources are processed into aggregated data and a given site evaluated the aggregated data using a predictive model to automatically determine whether the given site is likely to be a phishing site. The predictive model may be built using machine learning based on training data, e.g., including known phishing sites and/or known non-phishing sites. To determine whether an object corresponding to a site is likely a phishing-related object are described, various criteria are evaluated, including one or more features of the object when evaluated. The determination is output in some way, e.g., made available to a reputation service, used to block access to a site or warn a user before allowing access, and/or used to assist a hand grader in being more efficient in evaluating sites.