Disclosed herein are a method and apparatus that adaptively perform encoding and decoding based on image quality. An encoding apparatus may determine optimal Frames Per Second (FPS) for a video and may encode the video based on the determined FPS. Further, the encoding apparatus may provide improved temporal scalability. A decoding apparatus may select a frame to be displayed from among frames of a video depending on a required minimum satisfied user ratio. Through the selection of the frame, the decoding apparatus may provide improved temporal scalability.