A three-dimensional object detection device basically includes a three-dimensional object detection unit, a natural object assessment unit and a control unit. The three-dimensional object detection unit detects three-dimensional objects based on image information of a rear of a vehicle from a camera. The natural object assessment unit assesses that a detected three-dimensional object is a natural object, such as a plant or snow, based on an irregularity evaluation value calculated based on a first pixel number of first pixels representing a first predetermined differential in the differential image containing the detected three-dimensional object and a second pixel number of second pixels corresponding to the three-dimensional object and representing a second predetermined differential greater than the first predetermined differential.