For interlaced B-fields or interlaced B-frames, forward motion vectors are predicted by an encoder/decoder using forward motion vectors from a forward motion vector buffer, and backward motion vectors are predicted using backward motion vectors from a backward motion vector buffer. The resulting motion vectors are added to the corresponding buffer. Holes in motion vector buffers can be filled in with estimated motion vector values. An encoder/decoder switches prediction modes between fields in a field-coded macroblock of an interlaced B-frame. For interlaced B-frames and interlaced B-fields, an encoder/decoder computes direct mode motion vectors. For interlaced B-fields or interlaced B-frames, an encoder/decoder uses 4MV coding. An encoder/decoder uses "self-referencing" B-frames. An encoder sends binary information indicating whether a prediction mode is forward or not-forward for one or more macroblocks in an interfaced B-field. An encoder/decoder uses intra-coded B-fields ["BI-fields"].