An embodiment improves the operation of a H.264 and Joint Scalable Video Codec (e.g., JSVC/H.264 Amendment 3) video decoder by managing neighboring block data during the decoding process. An embodiment pre-computes neighboring block tables to efficiently locate the neighboring block data required to decode a current macroblock. In particular, the pre-computed most probable joint neighboring block tables disclosed herein handle both macroblock adaptive frame field (MBAFF) coding and non-MBAFF coding. An embodiment is further capable of managing variable block sizes. Other embodiments are described and claimed.