An online system receives host content items and supplementary content items to be presented in conjunction with one another. In embodiments, the online system receive an allocation of display instances of supplementary content items with host content items during a time interval. For a host content providers, the online system receives a count of display instances of supplementary content items in conjunction with host content items. The online system determines a content quality score based on user interactions with content from the host content provider and a count of display instances for host content items presented in conjunction with supplementary content items for a preceding time interval. The online system determines a content quality threshold based and, for host providers with content quality scored below the threshold, presents host content items in conjunction with supplementary content items to a user.