Described herein is a system and method for displaying a stereoscopic plenoptic image or video to a user, with eye-tracking capabilities. A display device displays a plenoptic image or video on a screen, and focuses the plenoptic image or video at the depth plane corresponding to the eye-coordinates of the user's gaze. The focused depth plane in the displayed plenoptic frame updates as the frames progress in the video stream, thereby constantly re-focusing the depth plane of the currently displayed plenoptic frame depending on the location of the user's gaze.