Disclosed is a method of encoding a direct mode when coding moving images, the method includes finding a prediction and a cost for each of a plurality of direct modes including a backward-based temporal direct mode and a forward-based temporal direct mode with respect to a current block to be encoded and determining one mode having the lowest cost as a result of comparing the costs as an optimal encoding mode, thereby performing encoding the direct mode.