Aspects of virtual field buffer based decoding are described. In one embodiment, a current picture field is read from a coded picture buffer comprising coded pictures of video, for example. The current picture may be associated with a top or bottom field polarity. According to decoding and picture output orders, the current picture field is assigned to an available field entry of a virtual field buffer, and an available frame index of a virtual frame map is assigned to the current picture field. The assignment of the available frame index to the current picture field is indicated to a decoder that decodes the current picture field with reference to the assigned frame index. According to aspects of the embodiments described herein, rather than determining complimentary picture fields after decoding, the assignment of complimentary picture fields to virtual frame indexes before decoding provides certain efficiencies.