In a video coding/decoding system, reference picture caches in a video coder and decoder may be partitioned dynamically based on camera and background motion can lead to improved coding efficiency and coding quality. When a camera is fixed and therefore exhibits low motion, a system may allocate larger portions of the reference picture cache to storage of long term reference frames. In this case, foreground elements of an image (for example, a person) may move in front of a relatively fixed background. Increasing the number of long term reference frames can increase the chances that, no matter where the foreground elements are within a frame currently being coded, the reference picture cache will contain at least one frame that provides an adequate prediction match to background elements within the new frame. Thus the background elements uncovered in the current frame can be coded at high quality with a low number of bits. When a camera exhibits high motion, the system may allocate larger portions of the reference picture cache to storage of short term reference frames.