Methods, systems, and media for color palette extraction for videos are provided. In some embodiments, the method comprises: identifying, at a server, a frame of a video content item; clustering pixels of the frame of the video content item based on a color of each of the pixels into a group of clusters; for each of a plurality of clusters in the group of clusters, determining an average color for the cluster; selecting a cluster of the plurality of clusters based on the average color of the cluster; determining a color palette corresponding to the frame of the video content item for one or more user interface elements in which the video content item is to be presented based on the average color of the selected cluster; and transmitting information indicating the color palette to a user device in response to a request to present the video content item.