Techniques are described for palette-based coding. In palette-based coding, a video coder may form a palette as a table of colors for representing video data of a given block. Palette-based coding may be useful for coding blocks of video data having a relatively small number of colors. Rather than coding actual pixel values or their residuals for the given block, the video coder may code index values for one or more of the pixels. The index values map the pixels to entries in the palette representing the colors of the pixels. Techniques are described for determining whether to disable filtering, such as deblocking filtering or sample adaptive offset (SAO) filtering, of palette coded blocks at a video encoder or a video decoder. Techniques are also described for modify a palette size and palette entries of a palette at the video encoder based on rate-distortion costs.