A system and method that optimizes a de-blocking filter process using local buffers. The local buffers may be off of the memory unit in which image data may be stored. The data associated with the image may be loaded into the local buffers. The data may be then fed into a filter, where the de-blocking filter process is executed, and the filtered data may be loaded back into the local buffers. The filtered data may then be written back to its original location in the memory.