A texture cache architecture facilitates access of compressed texture data in non-power of two formats, such as the Adaptive Scalable Texture Compression (ASTC) codec. In one implementation, the texture cache architecture includes a controller, a first buffer, a second buffer, and a texture decompressor. A first buffer stores one or more blocks of compressed texel data fetched, in response to a first request, from a first texture cache, where the one or more blocks of compressed texel data including at least requested texel data. The second buffer stores decompressed one or more blocks of compressed texel data and provides the decompressed requested texel data as output to a second texture cache. The one or more blocks of compressed texel data stored by the first buffer includes second texel data in addition to the requested texel data.