A method for real-time 2D to 3D video conversion includes receiving a decoded 2D video frame having an original resolution, downscaling the decoded 2D video frame into an associated 2D video frame having a lower resolution, and segmenting objects present in the downscaled 2D video frame into background objects and foreground objects. The method also includes generating a background depth map and a foreground depth map for the downscaled 2D video frame based on the segmented background and foreground objects, and deriving a frame depth map in the original resolution based on the background depth map and the foreground depth map. The method further includes providing a 3D video frame for display at a real-time playback rate. The 3D video frame is generated in the original resolution based on the frame depth map.