A system for video encoding is described herein. The system includes a processor to execute a multi-pass palette search and mapping on a video frame to generate palette candidates. The processor is to execute an intra block copy prediction on the video frame to generate intra-block-copy candidates. The processor is to also calculate a rate distortion optimization (RDO) cost for a set of generated residuals, the palette candidates, and the intra-block-copy candidates. The processor is to further also execute a final mode decision based on a comparison of the rate distortion optimization (RDO) costs.