An apparatus and method for writing and/or reading data in an image processing system are provided. The apparatus for writing data includes: a memory formed with at least one or more banks capable of being accessed independently; and a memory control unit controlling so that a luminance component, a first chrominance component and a second chrominance component belonging to one macroblock are written based on respective structures different to each other, in the memory. The apparatus for reading data includes: a memory formed with one or more banks capable of being accessed independently in which a luminance component, a first chrominance component, and a second chrominance component belonging to one macroblock are written based on respective structures different to each other, in each bank of the memory; and a memory control unit controlling so that a luminance component, a first chrominance component, and a second chrominance component required are read from the memory.