The technology described herein is directed to object compatibility-based identification and replacement of objects in digital representations of real-world environments for contextualized content delivery. In some implementations, an object compatibility and retargeting service that selects and analyzes a viewpoint (received from a user's client device) to identify objects that are the least compatible with other surrounding real-world objects in terms of style compatibility with the surrounding real-world objects and color compatibility with the background is described. The object compatibility and retargeting service also generates recommendations for replacing the least compatible object with objects/products having more style/design compatibility with the surrounding real-world objects and color compatibility with the background. Furthermore, the object compatibility and retargeting service can create personalized catalogues with the recommended objects/products embedded in the viewpoint in place of the least compatible object with similar pose and scale for retargeting the user.