An online system selects content items for presentation to viewing users of the online system based on a composite score associated with each content item that includes a quality component and a revenue component. The revenue component is based on a monetary amount an advertiser associated with the content item is willing to pay for each interaction with the content item by a prospective viewing user, while the quality component indicates the quality of the content item to the prospective viewing user. The quality component is predicted based on explicit user quality ratings received from viewing users for various content items previously presented to the viewing users, in which the viewing users have at least a threshold measure of similarity to the prospective viewing user and/or the various content items rated by the viewing users have at least a threshold measure of similarity to the content item being scored.