The invention relates to an optical field based image depth estimation method. The method mainly comprises three important parts of an optical field original data view point extraction method, a block matching based depth estimation algorithm and a remarkable feature constraint based depth optimization algorithm. The proposed optical field original data view point extraction method is used for performing view point separation on optical field data not subjected to demosaicing. The adopted block matching based depth estimation algorithm is used for performing similarity measurement on view points located in the same row or column as a central view point and corresponding blocks that record same light ray colors. For optimizing a depth estimation result, the invention proposes the remarkable feature constraint based depth optimization algorithm for extracting and matching remarkable feature points and taking a parallax error of the remarkable feature points as a strong constraint condition. The method avoids view point confusion caused in an interpolation process and improves depth estimation accuracy.